Methods and Systems for Controlling Cooperative Surgical Instruments

ABSTRACT

Systems, devices, and methods for controlling cooperative surgical instruments are provided. Various aspects of the present disclosure provide for coordinated operation of surgical instruments accessing various surgical sites from different or shared surgical approaches to achieve a common or cooperative surgical purpose. For example, various methods, devices, and systems disclosed herein can enable the coordinated treatment of tissue by disparate minimally invasive surgical systems that approach the tissue from varying anatomical spaces and must operate differently, but in concert with, one another to effect a desired surgical treatment.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 63/249,877, filed Sep. 29, 2021, and entitled “Methods and Systemsfor Controlling Cooperative Surgical Instruments,” the disclosure ofwhich is incorporated herein by reference in its entirety.

BACKGROUND

Some surgical procedures require the use of a plurality of surgicalinstruments operating on a region or portion of tissue at the same timeto successfully execute the procedure. In some situations, due to thenature of the procedure, it is necessary or helpful for the plurality ofsurgical instruments to cooperatively move and/or alter instrumentfunctionality based on the actions of one or more of the otherinstruments such that the instruments work collectively toward a commonsurgical goal.

However, it may be challenging to coordinate movement between multipleinstruments all at once while still maintaining appropriate focus onother aspects of a procedure. Additionally, direct visualization of andeasy access to instruments may be challenging or impossible, such asduring minimally invasive procedures.

Accordingly, there remains a need for improved methods and systems forcontrolling cooperative surgical instruments based on actions of theother surgical instruments to achieve a common surgical purpose.

SUMMARY

In an aspect, a system is provided herein with a first electrosurgicalinstrument, a second electrosurgical instrument, and a controller. Thefirst electrosurgical instrument has an end effector, and the endeffector is configured to grasp tissue and deliver energy thereto at afirst surgical treatment site located within a patient. The secondelectrosurgical instrument also has an end effector, and the endeffector is configured to grasp tissue and deliver energy thereto at thefirst surgical treatment site. The controller is configured to receivedata from the second electrosurgical instrument related to an amount ofenergy delivered by the first electrosurgical instrument to tissue, todetermine at least an amount of energy being delivered by the firstelectrosurgical instrument based on the data received, and to determinean adjustment of the amount of energy delivered by the firstelectrosurgical instrument based on the data received.

The system can have numerous variations. For example, the system canfurther include an endoscope system that has an image sensor configuredto capture image data characterizing an image of at least one of thefirst electrosurgical instrument or the second electrosurgicalinstrument. In another example, the controller can be configured tocompare the determined amount of energy being delivered by the firstelectrosurgical instrument to a predetermined threshold. In stillanother example, the controller can be configured to automaticallyadjust the amount of energy delivered by the first electrosurgicalinstrument based on the data received.

In some embodiments, the controller can be configured to provide analert regarding the amount of energy being delivered by the firstelectrosurgical instrument based on the data received. In some examples,the first electrosurgical instrument can be configure to operate ontissue at the first surgical treatment site from a first body cavity,and the second electrosurgical instrument can be configured to operateon tissue at the first surgical treatment site from a second bodycavity. In still other examples, the controller can be configured todetermine a tissue parameter of the tissue at the first surgicaltreatment site based on the data received. In some examples, thecontroller can be configured to change an energized state of the firstelectrosurgical instrument based on the data received.

In another aspect, a system is provided with a data processor and memorystoring instructions that are configured to cause the data processor toperform operations, including receiving, in real time, from a secondelectrosurgical instrument, data related to an amount of energydelivered by a first electrosurgical instrument to tissue at a firstsurgical treatment site located within a patient; determining, based onthe data received from the second electrosurgical instrument, at leastan amount of energy being delivered by the first electrosurgicalinstrument; and determining an adjustment of the amount of energydelivered by the first electrosurgical instrument based on the datareceived.

The system can have a number of different variations. For example, theoperations of the data processor can include receiving, in real time,from an endoscope system, an image of the first electrosurgicalinstrument. In another example, the operations of the data processor caninclude comparing the determined amount of energy being delivered by thefirst electrosurgical instrument to a predetermined threshold. In stillanother example, the operations of the data processor can includeautomatically adjusting the amount of energy delivered by the firstelectrosurgical instrument based on the data received. In still otherexamples, the operations of the data processor can include determining atissue parameter of the tissue at the first surgical treatment sitebased on the data received. In some examples, the operations of the atleast one data processor can further include changing an energized stateof the first electrosurgical instrument based on the data received.

In another aspect, a method is provided that includes receiving, by acontroller, in real time, from a second electrosurgical instrument, datarelated to an amount of energy delivered by the first electrosurgicalinstrument to tissue at a first surgical treatment site located within apatient. The method also includes determining, by the controller, atleast an amount of energy being delivered by the first electrosurgicalinstrument based on the data received, and determining an adjustment ofthe amount of energy delivered by the first electrosurgical instrumentbased on the data received.

The method can have numerous variations. For example, the method canfurther include receiving, in real time, from a first endoscope, imagesof the first electrosurgical instrument. In another example, the methodcan include automatically adjusting the amount of energy delivered bythe first electrosurgical instrument based on the data received. Instill another example, the method can include changing an energizedstate of the first electrosurgical instrument based on the datareceived.

In still another aspect, a system is provided that includes a firstsurgical instrument, a second surgical instrument, and a controller. Thefirst surgical instrument is configured to operate on target tissue at afirst surgical treatment site located within a patient, and the secondsurgical instrument is configured to anchor the first surgicalinstrument at the first surgical treatment site relative to the targettissue. The controller is configured to determine a total amount oftissue load being applied to the target tissue by the first and secondsurgical instruments, and to apply limits to tissue load applied by eachof the first and second surgical instruments to maintain the totalamount of tissue load on the target tissue below a predefined threshold.

The system can have numerous variations. For example, the first andsecond surgical instruments can be configured to be disposed within ashared body cavity. In another example, the first and second surgicalinstruments can be configured to be disposed on opposite sides of thetarget tissue in separate body cavities within the patient. In stillother examples, the first and second surgical instruments can beconfigured to capture tissue therebetween. In one example, the first andsecond surgical instruments can include surgical staplers,electrosurgical instruments, or graspers. In some embodiments, thesystem can include a flexible endoscope that has an image sensorconfigured to acquire an image of at least one of the first or secondsurgical instrument. In still other embodiments, the controller can beconfigured to receive the image from the flexible endoscope.

In still another aspect, a system is provided with a data processor andmemory storing instructions that are configured to cause the dataprocessor to perform operations. The operations include receiving, inreal time, data characterizing tissue load being applied by a firstsurgical instrument to target tissue at a first surgical site within apatient, and receiving, in real time, data characterizing tissue loadbeing applied by a second surgical instrument located within thepatient. Furthermore, the second surgical instrument is configured toanchor the first surgical instrument at the first surgical treatmentsite relative to the target tissue. The operations also includedetermining, based on at least the data received regarding the first andsecond surgical instruments, a total amount of tissue load being appliedto the target tissue by the first and second surgical instruments, andapplying limits to tissue load applied by each of the first and secondsurgical instruments on the target tissue to maintain the total amountof tissue load on the target tissue below a predefined threshold.

The system can have numerous different variations. For example, theoperations of the data processor can also include receiving, in realtime, from an image sensor of an endoscope, image data characterizing animage of at least one of the first or second surgical instruments. Inanother example, the operations of the data processor can includereceiving, in real time, from an image sensor of an endoscope, imagedata characterizing an image of at least one of the first or secondsurgical instruments. In some examples, the first and second surgicalinstruments are configured to be disposed within a shared body cavity.In other examples, the first and second surgical instruments can beconfigured to be disposed on opposite sides of the target tissue inseparate body cavities within the patient. In still other examples, thefirst and second surgical instruments are configured to capture tissuetherebetween. In one example, the first and second surgical instrumentscan include surgical staplers, electrosurgical instruments, or graspers.

In another aspect, a method is provided that includes receiving, at acontroller, in real time with performance of a surgical procedure on apatient, data regarding a first surgical instrument operating on targettissue at a first surgical treatment site within the patient. The methodalso includes receiving, at the controller, in real time withperformance of a surgical procedure on the patient, data regarding asecond surgical instrument that is anchoring the first surgicalinstrument at the first surgical treatment site relative to the targettissue. The method further includes determining, at the controller,based on at least the data received regarding the first and secondsurgical instruments, a total amount of tissue load being applied to thetarget tissue by the first and second surgical instruments, andapplying, using the controller, limits to tissue load applied by each ofthe first and second surgical instruments on the target tissue tomaintain the total amount of tissue load on the target tissue below apredefined threshold.

The method can have numerous variations. For example, the method caninclude altering, with the controller, the predefined threshold based ona location of the target tissue within the patient. In another example,the second surgical instrument can provide the data regarding the firstand second surgical instruments to a controller. In still anotherexample, the first and second surgical instruments can include surgicalstaplers, electrosurgical instruments, or graspers. In some examples,the method can also include receiving, at the controller, from aflexible endoscope having an image sensor, image data characterizing animage of at least one of the first or second surgical instruments.

In still another aspect, a system is provided with a first surgicalinstrument, a second surgical instrument, and a controller. The firstsurgical instrument is configured to operate on a first surgicaltreatment site located within a patient, and the second surgicalinstrument is configured to operate on the first surgical treatmentsite. The controller is configured to analyze movement of the firstsurgical instrument, and to automatically change movement of the secondsurgical instrument based on the analyzed movement of the first surgicalinstrument.

The system can have a number of different variations. For example, thechanged movement of the second surgical instrument can include at leastone of mimicked movement of the first surgical instrument and mirroredmovement of the first surgical instrument. In another example, theanalyzed movement of the first surgical instrument can include amonitored parameter of the first surgical instrument, and the controllercan be configured to automatically change movement of the secondsurgical instrument based on the monitored parameter. In still anotherexample the controller can be configured to automatically changemovement of the second surgical instrument to maintain the monitoredparameter at a consistent value. In further examples, the first andsecond surgical instruments can be configured to be disposed within ashared body cavity. In still other examples, the first and secondsurgical instruments can be configured to be disposed in separate bodycavities within the patient. In some examples, the first and secondsurgical instruments can include surgical staplers, electrosurgicalinstruments, or graspers. In still other examples, the system can alsoinclude a flexible endoscope having an image sensor configured tocapture image data characterizing an image of at least one of the firstor second surgical instruments.

In still another aspect, a system is provided that includes a dataprocessor and memory storing instructions that are configured to causethe at least one data processor to perform operations. The operationsinclude receiving, in real time, data characterizing movement of a firstsurgical instrument located within a patient, and determining, based onat least the received data, movement of the first surgical instrument,and automatically changing movement of a second surgical instrumentlocated within the patient based on the movement of the first surgicalinstrument.

The system can have numerous different variations. For example, thechanged movement of the second surgical instrument can include at leastone of mimicked or mirrored movement of the first surgical instrument.In some examples, the first and second surgical instruments can includesurgical staplers, electrosurgical instruments, or graspers. In otherexamples, operations of the at least one data processor can also includereceiving, in real time, from an image sensor of an endoscope, imagedata characterizing an image of at least one of the first or secondsurgical instruments.

In another aspect, a method is provided that includes receiving, at acontroller, in real time with performance of a surgical procedure on apatient, data characterizing movement of a first surgical instrumentlocated within a patient. The method also includes determining, at thecontroller, based on the received data, movement of the first surgicalinstrument, and automatically changing movement of a second surgicalinstrument located within the patient based on the movement of the firstsurgical instrument.

The method can have numerous variations. For example, the changedmovement of the second surgical instrument can include at least one ofmimicking or mirroring the determined movement of the first surgicalinstrument. In still another example, the method can include receiving,at the controller, in real time with the performance of the surgicalprocedure, from an image sensor of an endoscope, image datacharacterizing an image of at least one of the first or second surgicalinstruments.

BRIEF DESCRIPTION OF DRAWINGS

The present invention is described by way of reference to theaccompanying figures which are as follows:

FIG. 1 is a schematic view of one embodiment of a surgical visualizationsystem;

FIG. 2 is a schematic view of triangularization between a surgicaldevice, an imaging device, and a critical structure of FIG. 1 ;

FIG. 3 is a schematic view of another embodiment of a surgicalvisualization system;

FIG. 4 is a schematic view of one embodiment of a control system for asurgical visualization system;

FIG. 5 is a schematic view of one embodiment of a control circuit of acontrol system for a surgical visualization system;

FIG. 6 is a schematic view of one embodiment of a combinational logiccircuit of a surgical visualization system;

FIG. 7 is a schematic view of one embodiment of a sequential logiccircuit of a surgical visualization system;

FIG. 8 is a schematic view of yet another embodiment of a surgicalvisualization system;

FIG. 9 is a schematic view of another embodiment of a control system fora surgical visualization system;

FIG. 10 is a graph showing wavelength versus absorption coefficient forvarious biological materials;

FIG. 11 is a schematic view of one embodiment of a spectral emittervisualizing a surgical site;

FIG. 12 is a graph depicting illustrative hyperspectral identifyingsignatures to differentiate a ureter from obscurants;

FIG. 13 is a graph depicting illustrative hyperspectral identifyingsignatures to differentiate an artery from obscurants;

FIG. 14 is a graph depicting illustrative hyperspectral identifyingsignatures to differentiate a nerve from obscurants;

FIG. 15 is a schematic view of one embodiment of a near infrared (NIR)time-of-flight measurement system being utilized intraoperatively;

FIG. 16 shows a time-of-flight timing diagram for the system of FIG. 15;

FIG. 17 is a schematic view of another embodiment of a near infrared(NIR) time-of-flight measurement system being utilized intraoperatively;

FIG. 18 is a schematic view of one embodiment of a computer-implementedinteractive surgical system;

FIG. 19 is a schematic view of one embodiment a surgical system beingused to perform a surgical procedure in an operating room;

FIG. 20 is a schematic view of one embodiment of a surgical systemincluding a smart surgical instrument and a surgical hub;

FIG. 21 is a flowchart showing a method of controlling the smartsurgical instrument of FIG. 20 ;

FIG. 21A is a schematic view of a colon illustrating major resections ofthe colon;

FIG. 21B is a perspective partial cross-sectional view of one embodimentof a duodenal mucosal resurfacing procedure;

FIG. 22 is a schematic diagram of an exemplary surgical system that canprovide for cooperative control of surgical instruments;

FIG. 23 is an illustrative view of an example embodiment of surgicalinstruments that can be incorporated into the surgical system of FIG. 22;

FIG. 24 is an illustrative view of another example embodiment ofsurgical instruments that can be incorporated into the surgical systemof FIG. 22 ;

FIG. 25 is an illustrative view of exemplary surgical sites within apatient with another example embodiment of surgical instruments that canbe incorporated into the surgical system of FIG. 22 ;

FIG. 26 is another illustrative view of the exemplary surgical sites ofFIG. 25 ; and

FIG. 27 is a graph showing resistance load, displacement, and velocityof first and second instruments in which the second instrument'sresistance load, displacement, and velocity are dependent upon the firstinstrument.

DETAILED DESCRIPTION

Certain exemplary embodiments will now be described to provide anoverall understanding of the principles of the structure, function,manufacture, and use of the devices, systems, and methods disclosedherein. One or more examples of these embodiments are illustrated in theaccompanying drawings. A person skilled in the art will understand thatthe devices, systems, and methods specifically described herein andillustrated in the accompanying drawings are non-limiting exemplaryembodiments and that the scope of the present invention is definedsolely by the claims. The features illustrated or described inconnection with one exemplary embodiment may be combined with thefeatures of other embodiments. Such modifications and variations areintended to be included within the scope of the present invention.

Further, in the present disclosure, like-named components of theembodiments generally have similar features, and thus within aparticular embodiment each feature of each like-named component is notnecessarily fully elaborated upon. Additionally, to the extent thatlinear or circular dimensions are used in the description of thedisclosed systems, devices, and methods, such dimensions are notintended to limit the types of shapes that can be used in conjunctionwith such systems, devices, and methods. A person skilled in the artwill recognize that an equivalent to such linear and circular dimensionscan easily be determined for any geometric shape. A person skilled inthe art will appreciate that a dimension may not be a precise value butnevertheless be considered to be at about that value due to any numberof factors such as manufacturing tolerances and sensitivity ofmeasurement equipment. Sizes and shapes of the systems and devices, andthe components thereof, can depend at least on the size and shape ofcomponents with which the systems and devices will be used.

Surgical Visualization

In general, a surgical visualization system is configured to leverage“digital surgery” to obtain additional information about a patient'sanatomy and/or a surgical procedure. The surgical visualization systemis further configured to convey data to one or more medicalpractitioners in a helpful manner. Various aspects of the presentdisclosure provide improved visualization of the patient's anatomyand/or the surgical procedure, and/or use visualization to provideimproved control of a surgical tool (also referred to herein as a“surgical device” or a “surgical instrument”).

“Digital surgery” can embrace robotic systems, advanced imaging,advanced instrumentation, artificial intelligence, machine learning,data analytics for performance tracking and benchmarking, connectivityboth inside and outside of the operating room (OR), and more. Althoughvarious surgical visualization systems described herein can be used incombination with a robotic surgical system, surgical visualizationsystems are not limited to use with a robotic surgical system. Incertain instances, surgical visualization using a surgical visualizationsystem can occur without robotics and/or with limited and/or optionalrobotic assistance. Similarly, digital surgery can occur withoutrobotics and/or with limited and/or optional robotic assistance.

In certain instances, a surgical system that incorporates a surgicalvisualization system may enable smart dissection in order to identifyand avoid critical structures. Critical structures include anatomicalstructures such as a ureter, an artery such as a superior mesentericartery, a vein such as a portal vein, a nerve such as a phrenic nerve,and/or a tumor, among other anatomical structures. In other instances, acritical structure can be a foreign structure in the anatomical field,such as a surgical device, a surgical fastener, a clip, a tack, abougie, a band, a plate, and other foreign structures. Criticalstructures can be determined on a patient-by-patient and/or aprocedure-by-procedure basis. Smart dissection technology may provide,for example, improved intraoperative guidance for dissection and/or mayenable smarter decisions with critical anatomy detection and avoidancetechnology.

A surgical system incorporating a surgical visualization system mayenable smart anastomosis technologies that provide more consistentanastomoses at optimal location(s) with improved workflow. Cancerlocalization technologies may be improved with a surgical visualizationplatform. For example, cancer localization technologies can identify andtrack a cancer location, orientation, and its margins. In certaininstances, the cancer localization technologies may compensate formovement of a surgical instrument, a patient, and/or the patient'sanatomy during a surgical procedure in order to provide guidance back tothe point of interest for medical practitioner(s).

A surgical visualization system may provide improved tissuecharacterization and/or lymph node diagnostics and mapping. For example,tissue characterization technologies may characterize tissue type andhealth without the need for physical haptics, especially when dissectingand/or placing stapling devices within the tissue. Certain tissuecharacterization technologies may be utilized without ionizing radiationand/or contrast agents. With respect to lymph node diagnostics andmapping, a surgical visualization platform may, for example,preoperatively locate, map, and ideally diagnose the lymph system and/orlymph nodes involved in cancerous diagnosis and staging.

During a surgical procedure, information available to a medicalpractitioner via the “naked eye” and/or an imaging system may provide anincomplete view of the surgical site. For example, certain structures,such as structures embedded or buried within an organ, can be at leastpartially concealed or hidden from view. Additionally, certaindimensions and/or relative distances can be difficult to ascertain withexisting sensor systems and/or difficult for the “naked eye” toperceive. Moreover, certain structures can move pre-operatively (e.g.,before a surgical procedure but after a preoperative scan) and/orintraoperatively. In such instances, the medical practitioner can beunable to accurately determine the location of a critical structureintraoperatively.

When the position of a critical structure is uncertain and/or when theproximity between the critical structure and a surgical tool is unknown,a medical practitioner's decision-making process can be inhibited. Forexample, a medical practitioner may avoid certain areas in order toavoid inadvertent dissection of a critical structure; however, theavoided area may be unnecessarily large and/or at least partiallymisplaced. Due to uncertainty and/or overly/excessive exercises incaution, the medical practitioner may not access certain desiredregions. For example, excess caution may cause a medical practitioner toleave a portion of a tumor and/or other undesirable tissue in an effortto avoid a critical structure even if the critical structure is not inthe particular area and/or would not be negatively impacted by themedical practitioner working in that particular area. In certaininstances, surgical results can be improved with increased knowledgeand/or certainty, which can allow a surgeon to be more accurate and, incertain instances, less conservative/more aggressive with respect toparticular anatomical areas.

A surgical visualization system can allow for intraoperativeidentification and avoidance of critical structures. The surgicalvisualization system may thus enable enhanced intraoperative decisionmaking and improved surgical outcomes. The surgical visualization systemcan provide advanced visualization capabilities beyond what a medicalpractitioner sees with the “naked eye” and/or beyond what an imagingsystem can recognize and/or convey to the medical practitioner. Thesurgical visualization system can augment and enhance what a medicalpractitioner is able to know prior to tissue treatment (e.g.,dissection, etc.) and, thus, may improve outcomes in various instances.As a result, the medical practitioner can confidently maintain momentumthroughout the surgical procedure knowing that the surgicalvisualization system is tracking a critical structure, which may beapproached during dissection, for example. The surgical visualizationsystem can provide an indication to the medical practitioner insufficient time for the medical practitioner to pause and/or slow downthe surgical procedure and evaluate the proximity to the criticalstructure to prevent inadvertent damage thereto. The surgicalvisualization system can provide an ideal, optimized, and/orcustomizable amount of information to the medical practitioner to allowthe medical practitioner to move confidently and/or quickly throughtissue while avoiding inadvertent damage to healthy tissue and/orcritical structure(s) and, thus, to minimize the risk of harm resultingfrom the surgical procedure.

Surgical visualization systems are described in detail below. Ingeneral, a surgical visualization system can include a first lightemitter configured to emit a plurality of spectral waves, a second lightemitter configured to emit a light pattern, and a receiver, or sensor,configured to detect visible light, molecular responses to the spectralwaves (spectral imaging), and/or the light pattern. The surgicalvisualization system can also include an imaging system and a controlcircuit in signal communication with the receiver and the imagingsystem. Based on output from the receiver, the control circuit candetermine a geometric surface map, e.g., three-dimensional surfacetopography, of the visible surfaces at the surgical site and a distancewith respect to the surgical site, such as a distance to an at leastpartially concealed structure. The imaging system can convey thegeometric surface map and the distance to a medical practitioner. Insuch instances, an augmented view of the surgical site provided to themedical practitioner can provide a representation of the concealedstructure within the relevant context of the surgical site. For example,the imaging system can virtually augment the concealed structure on thegeometric surface map of the concealing and/or obstructing tissuesimilar to a line drawn on the ground to indicate a utility line belowthe surface. Additionally or alternatively, the imaging system canconvey the proximity of a surgical tool to the visible and obstructingtissue and/or to the at least partially concealed structure and/or adepth of the concealed structure below the visible surface of theobstructing tissue. For example, the visualization system can determinea distance with respect to the augmented line on the surface of thevisible tissue and convey the distance to the imaging system.

Throughout the present disclosure, any reference to “light,” unlessspecifically in reference to visible light, can include electromagneticradiation (EMR) or photons in the visible and/or non-visible portions ofthe EMR wavelength spectrum. The visible spectrum, sometimes referred toas the optical spectrum or luminous spectrum, is that portion of theelectromagnetic spectrum that is visible to (e.g., can be detected by)the human eye and may be referred to as “visible light” or simply“light.” A typical human eye will respond to wavelengths in air that arefrom about 380 nm to about 750 nm. The invisible spectrum (e.g., thenon-luminous spectrum) is that portion of the electromagnetic spectrumthat lies below and above the visible spectrum. The invisible spectrumis not detectable by the human eye. Wavelengths greater than about 750nm are longer than the red visible spectrum, and they become invisibleinfrared (IR), microwave, and radio electromagnetic radiation.Wavelengths less than about 380 nm are shorter than the violet spectrum,and they become invisible ultraviolet, x-ray, and gamma rayelectromagnetic radiation.

FIG. 1 illustrates one embodiment of a surgical visualization system100. The surgical visualization system 100 is configured to create avisual representation of a critical structure 101 within an anatomicalfield. The critical structure 101 can include a single criticalstructure or a plurality of critical structures. As discussed herein,the critical structure 101 can be any of a variety of structures, suchas an anatomical structure, e.g., a ureter, an artery such as a superiormesenteric artery, a vein such as a portal vein, a nerve such as aphrenic nerve, a vessel, a tumor, or other anatomical structure, or aforeign structure, e.g., a surgical device, a surgical fastener, asurgical clip, a surgical tack, a bougie, a surgical band, a surgicalplate, or other foreign structure. As discussed herein, the criticalstructure 101 can be identified on a patient-by-patient and/or aprocedure-by-procedure basis. Embodiments of critical structures and ofidentifying critical structures using a visualization system are furtherdescribed in U.S. Pat. No. 10,792,034 entitled “Visualization OfSurgical Devices” issued Oct. 6, 2020, which is hereby incorporated byreference in its entirety.

In some instances, the critical structure 101 can be embedded in tissue103. The tissue 103 can be any of a variety of tissues, such as fat,connective tissue, adhesions, and/or organs. Stated differently, thecritical structure 101 may be positioned below a surface 105 of thetissue 103. In such instances, the tissue 103 conceals the criticalstructure 101 from the medical practitioner's “naked eye” view. Thetissue 103 also obscures the critical structure 101 from the view of animaging device 120 of the surgical visualization system 100. Instead ofbeing fully obscured, the critical structure 101 can be partiallyobscured from the view of the medical practitioner and/or the imagingdevice 120.

The surgical visualization system 100 can be used for clinical analysisand/or medical intervention. In certain instances, the surgicalvisualization system 100 can be used intraoperatively to providereal-time information to the medical practitioner during a surgicalprocedure, such as real-time information regarding proximity data,dimensions, and/or distances. A person skilled in the art willappreciate that information may not be precisely real time butnevertheless be considered to be real time for any of a variety ofreasons, such as time delay induced by data transmission, time delayinduced by data processing, and/or sensitivity of measurement equipment.The surgical visualization system 100 is configured for intraoperativeidentification of critical structure(s) and/or to facilitate theavoidance of the critical structure(s) 101 by a surgical device. Forexample, by identifying the critical structure 101, a medicalpractitioner can avoid maneuvering a surgical device around the criticalstructure 101 and/or a region in a predefined proximity of the criticalstructure 101 during a surgical procedure. For another example, byidentifying the critical structure 101, a medical practitioner can avoiddissection of and/or near the critical structure 101, thereby helping toprevent damage to the critical structure 101 and/or helping to prevent asurgical device being used by the medical practitioner from beingdamaged by the critical structure 101.

The surgical visualization system 100 is configured to incorporatetissue identification and geometric surface mapping in combination withthe surgical visualization system's distance sensor system 104. Incombination, these features of the surgical visualization system 100 candetermine a position of a critical structure 101 within the anatomicalfield and/or the proximity of a surgical device 102 to the surface 105of visible tissue 103 and/or to the critical structure 101. Moreover,the surgical visualization system 100 includes an imaging system thatincludes the imaging device 120 configured to provide real-time views ofthe surgical site. The imaging device 120 can include, for example, aspectral camera (e.g., a hyperspectral camera, multispectral camera, orselective spectral camera), which is configured to detect reflectedspectral waveforms and generate a spectral cube of images based on themolecular response to the different wavelengths. Views from the imagingdevice 120 can be provided in real time to a medical practitioner, suchas on a display (e.g., a monitor, a computer tablet screen, etc.). Thedisplayed views can be augmented with additional information based onthe tissue identification, landscape mapping, and the distance sensorsystem 104. In such instances, the surgical visualization system 100includes a plurality of subsystems—an imaging subsystem, a surfacemapping subsystem, a tissue identification subsystem, and/or a distancedetermining subsystem. These subsystems can cooperate tointra-operatively provide advanced data synthesis and integratedinformation to the medical practitioner.

The imaging device 120 can be configured to detect visible light,spectral light waves (visible or invisible), and a structured lightpattern (visible or invisible). Examples of the imaging device 120includes scopes, e.g., an endoscope, an arthroscope, an angioscope, abronchoscope, a choledochoscope, a colonoscope, a cytoscope, aduodenoscope, an enteroscope, an esophagogastro-duodenoscope(gastroscope), a laryngoscope, a nasopharyngo-neproscope, asigmoidoscope, a thoracoscope, an ureteroscope, or an exoscope. Scopescan be particularly useful in minimally invasive surgical procedures. Inopen surgery applications, the imaging device 120 may not include ascope.

The tissue identification subsystem can be achieved with a spectralimaging system. The spectral imaging system can rely on imaging such ashyperspectral imaging, multispectral imaging, or selective spectralimaging. Embodiments of hyperspectral imaging of tissue are furtherdescribed in U.S. Pat. No. 9,274,047 entitled “System And Method ForGross Anatomic Pathology Using Hyperspectral Imaging” issued Mar. 1,2016, which is hereby incorporated by reference in its entirety.

The surface mapping subsystem can be achieved with a light patternsystem. Various surface mapping techniques using a light pattern (orstructured light) for surface mapping can be utilized in the surgicalvisualization systems described herein. Structured light is the processof projecting a known pattern (often a grid or horizontal bars) on to asurface. In certain instances, invisible (or imperceptible) structuredlight can be utilized, in which the structured light is used withoutinterfering with other computer vision tasks for which the projectedpattern may be confusing. For example, infrared light or extremely fastframe rates of visible light that alternate between two exact oppositepatterns can be utilized to prevent interference. Embodiments of surfacemapping and a surgical system including a light source and a projectorfor projecting a light pattern are further described in U.S. Pat. Pub.No. 2017/0055819 entitled “Set Comprising A Surgical Instrument”published Mar. 2, 2017, U.S. Pat. Pub. No. 2017/0251900 entitled“Depiction System” published Sep. 7, 2017, and U.S. patent applicationSer. No. 16/729,751 entitled “Surgical Systems For Generating ThreeDimensional Constructs Of Anatomical Organs And Coupling IdentifiedAnatomical Structures Thereto” filed Dec. 30, 2019, which are herebyincorporated by reference in their entireties.

The distance determining system can be incorporated into the surfacemapping system. For example, structured light can be utilized togenerate a three-dimensional (3D) virtual model of the visible surface105 and determine various distances with respect to the visible surface105. Additionally or alternatively, the distance determining system canrely on time-of-flight measurements to determine one or more distancesto the identified tissue (or other structures) at the surgical site.

The surgical visualization system 100 also includes a surgical device102. The surgical device 102 can be any suitable surgical device.Examples of the surgical device 102 includes a surgical dissector, asurgical stapler, a surgical grasper, a clip applier, a smoke evacuator,a surgical energy device (e.g., mono-polar probes, bi-polar probes,ablation probes, an ultrasound device, an ultrasonic end effector,etc.), etc. In some embodiments, the surgical device 102 includes an endeffector having opposing jaws that extend from a distal end of a shaftof the surgical device 102 and that are configured to engage tissuetherebetween.

The surgical visualization system 100 can be configured to identify thecritical structure 101 and a proximity of the surgical device 102 to thecritical structure 101. The imaging device 120 of the surgicalvisualization system 100 is configured to detect light at variouswavelengths, such as visible light, spectral light waves (visible orinvisible), and a structured light pattern (visible or invisible). Theimaging device 120 can include a plurality of lenses, sensors, and/orreceivers for detecting the different signals. For example, the imagingdevice 120 can be a hyperspectral, multispectral, or selective spectralcamera, as described herein. The imaging device 120 can include awaveform sensor 122 (such as a spectral image sensor, detector, and/orthree-dimensional camera lens). For example, the imaging device 120 caninclude a right-side lens and a left-side lens used together to recordtwo two-dimensional images at the same time and, thus, generate athree-dimensional (3D) image of the surgical site, render athree-dimensional image of the surgical site, and/or determine one ormore distances at the surgical site. Additionally or alternatively, theimaging device 120 can be configured to receive images indicative of thetopography of the visible tissue and the identification and position ofhidden critical structures, as further described herein. For example, afield of view of the imaging device 120 can overlap with a pattern oflight (structured light) on the surface 105 of the tissue 103, as shownin FIG. 1 .

As in this illustrated embodiment, the surgical visualization system 100can be incorporated into a robotic surgical system 110. The roboticsurgical system 110 can have a variety of configurations, as discussedherein. In this illustrated embodiment, the robotic surgical system 110includes a first robotic arm 112 and a second robotic arm 114. Therobotic arms 112, 114 each include rigid structural members 116 andjoints 118, which can include servomotor controls. The first robotic arm112 is configured to maneuver the surgical device 102, and the secondrobotic arm 114 is configured to maneuver the imaging device 120. Arobotic control unit of the robotic surgical system 110 is configured toissue control motions to the first and second robotic arms 112, 114,which can affect the surgical device 102 and the imaging device 120,respectively.

In some embodiments, one or more of the robotic arms 112, 114 can beseparate from the main robotic system 110 used in the surgicalprocedure. For example, at least one of the robotic arms 112, 114 can bepositioned and registered to a particular coordinate system without aservomotor control. For example, a closed-loop control system and/or aplurality of sensors for the robotic arms 112, 114 can control and/orregister the position of the robotic arm(s) 112, 114 relative to theparticular coordinate system. Similarly, the position of the surgicaldevice 102 and the imaging device 120 can be registered relative to aparticular coordinate system.

Examples of robotic surgical systems include the Ottava™robotic-assisted surgery system (Johnson & Johnson of New Brunswick,N.J.), da Vinci® surgical systems (Intuitive Surgical, Inc. ofSunnyvale, Calif.), the Hugo™ robotic-assisted surgery system (MedtronicPLC of Minneapolis, Minn.), the Versius® surgical robotic system (CMRSurgical Ltd of Cambridge, UK), and the Monarch® platform (Auris Health,Inc. of Redwood City, Calif.). Embodiments of various robotic surgicalsystems and using robotic surgical systems are further described in U.S.Pat. Pub. No. 2018/0177556 entitled “Flexible Instrument Insertion UsingAn Adaptive Force Threshold” filed Dec. 28, 2016, U.S. Pat. Pub. No.2020/0000530 entitled “Systems And Techniques For Providing MultiplePerspectives During Medical Procedures” filed Apr. 16, 2019, U.S. Pat.Pub. No. 2020/0170720 entitled “Image-Based Branch Detection And MappingFor Navigation” filed Feb. 7, 2020, U.S. Pat. Pub. No. 2020/0188043entitled “Surgical Robotics System” filed Dec. 9, 2019, U.S. Pat. Pub.No. 2020/0085516 entitled “Systems And Methods For Concomitant MedicalProcedures” filed Sep. 3, 2019, U.S. Pat. No. 8,831,782 entitled“Patient-Side Surgeon Interface For A Teleoperated Surgical Instrument”filed Jul. 15, 2013, and Intl. Pat. Pub. No. WO 2014151621 entitled“Hyperdexterous Surgical System” filed Mar. 13, 2014, which are herebyincorporated by reference in their entireties.

The surgical visualization system 100 also includes an emitter 106. Theemitter 106 is configured to emit a pattern of light, such as stripes,grid lines, and/or dots, to enable the determination of the topographyor landscape of the surface 105. For example, projected light arrays 130can be used for three-dimensional scanning and registration on thesurface 105. The projected light arrays 130 can be emitted from theemitter 106 located on the surgical device 102 and/or one of the roboticarms 112, 114 and/or the imaging device 120. In one aspect, theprojected light array 130 is employed by the surgical visualizationsystem 100 to determine the shape defined by the surface 105 of thetissue 103 and/or motion of the surface 105 intraoperatively. Theimaging device 120 is configured to detect the projected light arrays130 reflected from the surface 105 to determine the topography of thesurface 105 and various distances with respect to the surface 105.

As in this illustrated embodiment, the imaging device 120 can include anoptical waveform emitter 123, such as by being mounted on or otherwiseattached on the imaging device 120. The optical waveform emitter 123 isconfigured to emit electromagnetic radiation 124 (near-infrared (NIR)photons) that can penetrate the surface 105 of the tissue 103 and reachthe critical structure 101. The imaging device 120 and the opticalwaveform emitter 123 can be positionable by the robotic arm 114. Theoptical waveform emitter 123 is mounted on or otherwise on the imagingdevice 122 but in other embodiments can be positioned on a separatesurgical device from the imaging device 120. A corresponding waveformsensor 122 (e.g., an image sensor, spectrometer, or vibrational sensor)of the imaging device 120 is configured to detect the effect of theelectromagnetic radiation received by the waveform sensor 122. Thewavelengths of the electromagnetic radiation 124 emitted by the opticalwaveform emitter 123 are configured to enable the identification of thetype of anatomical and/or physical structure, such as the criticalstructure 101. The identification of the critical structure 101 can beaccomplished through spectral analysis, photo-acoustics, and/orultrasound, for example. In one aspect, the wavelengths of theelectromagnetic radiation 124 can be variable. The waveform sensor 122and optical waveform emitter 123 can be inclusive of a multispectralimaging system and/or a selective spectral imaging system, for example.In other instances, the waveform sensor 122 and optical waveform emitter123 can be inclusive of a photoacoustic imaging system, for example.

The distance sensor system 104 of the surgical visualization system 100is configured to determine one or more distances at the surgical site.The distance sensor system 104 can be a time-of-flight distance sensorsystem that includes an emitter, such as the emitter 106 as in thisillustrated embodiment, and that includes a receiver 108. In otherinstances, the time-of-flight emitter can be separate from thestructured light emitter. The emitter 106 can include a very tiny lasersource, and the receiver 108 can include a matching sensor. The distancesensor system 104 is configured to detect the “time of flight,” or howlong the laser light emitted by the emitter 106 has taken to bounce backto the sensor portion of the receiver 108. Use of a very narrow lightsource in the emitter 106 enables the distance sensor system 104 todetermining the distance to the surface 105 of the tissue 103 directlyin front of the distance sensor system 104.

The receiver 108 of the distance sensor system 104 is positioned on thesurgical device 102 in this illustrated embodiment, but in otherembodiments the receiver 108 can be mounted on a separate surgicaldevice instead of the surgical device 102. For example, the receiver 108can be mounted on a cannula or trocar through which the surgical device102 extends to reach the surgical site. In still other embodiments, thereceiver 108 for the distance sensor system 104 can be mounted on aseparate robotically-controlled arm of the robotic system 110 (e.g., onthe second robotic arm 114) than the first robotic arm 112 to which thesurgical device 102 is coupled, can be mounted on a movable arm that isoperated by another robot, or be mounted to an operating room (OR) tableor fixture. In some embodiments, the imaging device 120 includes thereceiver 108 to allow for determining the distance from the emitter 106to the surface 105 of the tissue 103 using a line between the emitter106 on the surgical device 102 and the imaging device 120. For example,the distance d_(e) can be triangulated based on known positions of theemitter 106 (on the surgical device 102) and the receiver 108 (on theimaging device 120) of the distance sensor system 104. Thethree-dimensional position of the receiver 108 can be known and/orregistered to the robot coordinate plane intraoperatively.

As in this illustrated embodiment, the position of the emitter 106 ofthe distance sensor system 104 can be controlled by the first roboticarm 112, and the position of the receiver 108 of the distance sensorsystem 104 can be controlled by the second robotic arm 114. In otherembodiments, the surgical visualization system 100 can be utilized apartfrom a robotic system. In such instances, the distance sensor system 104can be independent of the robotic system.

In FIG. 1 , d_(e) is emitter-to-tissue distance from the emitter 106 tothe surface 105 of the tissue 103, and d_(t) is device-to-tissuedistance from a distal end of the surgical device 102 to the surface 105of the tissue 103. The distance sensor system 104 is configured todetermine the emitter-to-tissue distance d_(e). The device-to-tissuedistance d_(t) is obtainable from the known position of the emitter 106on the surgical device 102, e.g., on a shaft thereof proximal to thesurgical device's distal end, relative to the distal end of the surgicaldevice 102. In other words, when the distance between the emitter 106and the distal end of the surgical device 102 is known, thedevice-to-tissue distance d_(t) can be determined from theemitter-to-tissue distance d_(e). In some embodiments, the shaft of thesurgical device 102 can include one or more articulation joints and canbe articulatable with respect to the emitter 106 and jaws at the distalend of the surgical device 102. The articulation configuration caninclude a multi-joint vertebrae-like structure, for example. In someembodiments, a three-dimensional camera can be utilized to triangulateone or more distances to the surface 105.

In FIG. 1 , d_(w) is camera-to-critical structure distance from theoptical waveform emitter 123 located on the imaging device 120 to thesurface of the critical structure 101, and d_(A) is a depth of thecritical structure 101 below the surface 105 of the tissue 103 (e.g.,the distance between the portion of the surface 105 closest to thesurgical device 102 and the critical structure 101). The time-of-flightof the optical waveforms emitted from the optical waveform emitter 123located on the imaging device 120 are configured to determine thecamera-to-critical structure distance d_(w).

As shown in FIG. 2 , the depth d_(A) of the critical structure 101relative to the surface 105 of the tissue 103 can be determined bytriangulating from the distance d_(w) and known positions of the emitter106 on the surgical device 102 and the optical waveform emitter 123 onthe imaging device 120 (and, thus, the known distance d_(x)therebetween) to determine the distance d_(y), which is the sum of thedistances d_(e) and d_(A). Additionally or alternatively, time-of-flightfrom the optical waveform emitter 123 can be configured to determine thedistance from the optical waveform emitter 123 to the surface 105 of thetissue 103. For example, a first waveform (or range of waveforms) can beutilized to determine the camera-to-critical structure distance d_(w)and a second waveform (or range of waveforms) can be utilized todetermine the distance to the surface 105 of the tissue 103. In suchinstances, the different waveforms can be utilized to determine thedepth of the critical structure 101 below the surface 105 of the tissue103.

Additionally or alternatively, the distance d_(A) can be determined froman ultrasound, a registered magnetic resonance imaging (MRI), orcomputerized tomography (CT) scan. In still other instances, thedistance d_(A) can be determined with spectral imaging because thedetection signal received by the imaging device 120 can vary based onthe type of material, e.g., type of the tissue 103. For example, fat candecrease the detection signal in a first way, or a first amount, andcollagen can decrease the detection signal in a different, second way,or a second amount.

In another embodiment of a surgical visualization system 160 illustratedin FIG. 3 , a surgical device 162, and not the imaging device 120,includes the optical waveform emitter 123 and the waveform sensor 122that is configured to detect the reflected waveforms. The opticalwaveform emitter 123 is configured to emit waveforms for determining thedistances d_(t) and d_(w) from a common device, such as the surgicaldevice 162, as described herein. In such instances, the distance d_(A)from the surface 105 of the tissue 103 to the surface of the criticalstructure 101 can be determined as follows:

d _(A) =d _(w) −d _(t)

The surgical visualization system 100 includes a control systemconfigured to control various aspects of the surgical visualizationsystem 100. FIG. 4 illustrates one embodiment of a control system 133that can be utilized as the control system of the surgical visualizationsystem 100 (or other surgical visualization system described herein).The control system 133 includes a control circuit 132 configured to bein signal communication with a memory 134. The memory 134 is configuredto store instructions executable by the control circuit 132, such asinstructions to determine and/or recognize critical structures (e.g.,the critical structure 101 of FIG. 1 ), instructions to determine and/orcompute one or more distances and/or three-dimensional digitalrepresentations, and instructions to communicate information to amedical practitioner. As in this illustrated embodiment, the memory 134can store surface mapping logic 136, imaging logic 138, tissueidentification logic 140, and distance determining logic 141, althoughthe memory 134 can store any combinations of the logics 136, 138, 140,141 and/or can combine various logics together. The control system 133also includes an imaging system 142 including a camera 144 (e.g., theimaging system including the imaging device 120 of FIG. 1 ), a display146 (e.g., a monitor, a computer tablet screen, etc.), and controls 148of the camera 144 and the display 146. The camera 144 includes an imagesensor 135 (e.g., the waveform sensor 122) configured to receive signalsfrom various light sources emitting light at various visible andinvisible spectra (e.g., visible light, spectral imagers,three-dimensional lens, etc.). The display 146 is configured to depictreal, virtual, and/or virtually-augmented images and/or information to amedical practitioner.

In an exemplary embodiment, the image sensor 135 is a solid-stateelectronic device containing up to millions of discrete photodetectorsites called pixels. The image sensor 135 technology falls into one oftwo categories: Charge-Coupled Device (CCD) and Complementary MetalOxide Semiconductor (CMOS) imagers and more recently, short-waveinfrared (SWIR) is an emerging technology in imaging. Another type ofthe image sensor 135 employs a hybrid CCD/CMOS architecture (sold underthe name “sCMOS”) and consists of CMOS readout integrated circuits(ROICs) that are bump bonded to a CCD imaging substrate. CCD and CMOSimage sensors 135 are sensitive to wavelengths in a range of about 350nm to about 1050 nm, such as in a range of about 400 nm to about 1000nm. A person skilled in the art will appreciate that a value may not beprecisely at a value but nevertheless considered to be about that valuefor any of a variety of reasons, such as sensitivity of measurementequipment and manufacturing tolerances. CMOS sensors are, in general,more sensitive to IR wavelengths than CCD sensors. Solid state imagesensors 135 are based on the photoelectric effect and, as a result,cannot distinguish between colors. Accordingly, there are two types ofcolor CCD cameras: single chip and three-chip. Single chip color CCDcameras offer a common, low-cost imaging solution and use a mosaic(e.g., Bayer) optical filter to separate incoming light into a series ofcolors and employ an interpolation algorithm to resolve full colorimages. Each color is, then, directed to a different set of pixels.Three-chip color CCD cameras provide higher resolution by employing aprism to direct each section of the incident spectrum to a differentchip. More accurate color reproduction is possible, as each point inspace of the object has separate RGB intensity values, rather than usingan algorithm to determine the color. Three-chip cameras offer extremelyhigh resolutions.

The control system 133 also includes an emitter (e.g., the emitter 106)including a spectral light source 150 and a structured light source 152each operably coupled to the control circuit 133. A single source can bepulsed to emit wavelengths of light in the spectral light source 150range and wavelengths of light in the structured light source 152 range.Alternatively, a single light source can be pulsed to provide light inthe invisible spectrum (e.g., infrared spectral light) and wavelengthsof light on the visible spectrum. The spectral light source 150 can be,for example, a hyperspectral light source, a multispectral light source,and/or a selective spectral light source. The tissue identificationlogic 140 is configured to identify critical structure(s) (e.g., thecritical structure 101 of FIG. 1 ) via data from the spectral lightsource 150 received by the image sensor 135 of the camera 144. Thesurface mapping logic 136 is configured to determine the surfacecontours of the visible tissue (e.g., the tissue 103) based on reflectedstructured light. With time-of-flight measurements, the distancedetermining logic 141 is configured to determine one or more distance(s)to the visible tissue and/or the critical structure. Output from each ofthe surface mapping logic 136, the tissue identification logic 140, andthe distance determining logic 141 is configured to be provided to theimaging logic 138, and combined, blended, and/or overlaid by the imaginglogic 138 to be conveyed to a medical practitioner via the display 146of the imaging system 142.

The control circuit 132 can have a variety of configurations. FIG. 5illustrates one embodiment of a control circuit 170 that can be used asthe control circuit 132 configured to control aspects of the surgicalvisualization system 100. The control circuit 170 is configured toimplement various processes described herein. The control circuit 170includes a microcontroller that includes a processor 172 (e.g., amicroprocessor or microcontroller) operably coupled to a memory 174. Thememory 174 is configured to store machine-executable instructions that,when executed by the processor 172, cause the processor 172 to executemachine instructions to implement various processes described herein.The processor 172 can be any one of a number of single-core or multicoreprocessors known in the art. The memory 174 can include volatile andnon-volatile storage media. The processor 172 includes an instructionprocessing unit 176 and an arithmetic unit 178. The instructionprocessing unit 176 is configured to receive instructions from thememory 174.

The surface mapping logic 136, the imaging logic 138, the tissueidentification logic 140, and the distance determining logic 141 canhave a variety of configurations. FIG. 6 illustrates one embodiment of acombinational logic circuit 180 configured to control aspects of thesurgical visualization system 100 using logic such as one or more of thesurface mapping logic 136, the imaging logic 138, the tissueidentification logic 140, and the distance determining logic 141. Thecombinational logic circuit 180 includes a finite state machine thatincludes a combinational logic 182 configured to receive data associatedwith a surgical device (e.g. the surgical device 102 and/or the imagingdevice 120) at an input 184, process the data by the combinational logic182, and provide an output 184 to a control circuit (e.g., the controlcircuit 132).

FIG. 7 illustrates one embodiment of a sequential logic circuit 190configured to control aspects of the surgical visualization system 100using logic such as one or more of the surface mapping logic 136, theimaging logic 138, the tissue identification logic 140, and the distancedetermining logic 141. The sequential logic circuit 190 includes afinite state machine that includes a combinational logic 192, a memory194, and a clock 196. The memory 194 is configured to store a currentstate of the finite state machine. The sequential logic circuit 190 canbe synchronous or asynchronous. The combinational logic 192 isconfigured to receive data associated with a surgical device (e.g. thesurgical device 102 and/or the imaging device 120) at an input 426,process the data by the combinational logic 192, and provide an output499 to a control circuit (e.g., the control circuit 132). In someembodiments, the sequential logic circuit 190 can include a combinationof a processor (e.g., processor 172 of FIG. 5 ) and a finite statemachine to implement various processes herein. In some embodiments, thefinite state machine can include a combination of a combinational logiccircuit (e.g., the combinational logic circuit 192 of FIG. 7 ) and thesequential logic circuit 190.

FIG. 8 illustrates another embodiment of a surgical visualization system200. The surgical visualization system 200 is generally configured andused similar to the surgical visualization system 100 of FIG. 1 , e.g.,includes a surgical device 202 and an imaging device 220. The imagingdevice 220 includes a spectral light emitter 223 configured to emitspectral light in a plurality of wavelengths to obtain a spectral imageof hidden structures, for example. The imaging device 220 can alsoinclude a three-dimensional camera and associated electronic processingcircuits. The surgical visualization system 200 is shown being utilizedintraoperatively to identify and facilitate avoidance of certaincritical structures, such as a ureter 201 a and vessels 201 b, in anorgan 203 (a uterus in this embodiment) that are not visible on asurface 205 of the organ 203.

The surgical visualization system 200 is configured to determine anemitter-to-tissue distance d_(e) from an emitter 206 on the surgicaldevice 202 to the surface 205 of the uterus 203 via structured light.The surgical visualization system 200 is configured to extrapolate adevice-to-tissue distance d_(t) from the surgical device 202 to thesurface 205 of the uterus 203 based on the emitter-to-tissue distanced_(e). The surgical visualization system 200 is also configured todetermine a tissue-to-ureter distance d_(A) from the ureter 201 a to thesurface 205 and a camera-to ureter distance d_(w) from the imagingdevice 220 to the ureter 201 a. As described herein, e.g., with respectto the surgical visualization system 100 of FIG. 1 , the surgicalvisualization system 200 is configured to determine the distance d_(w)with spectral imaging and time-of-flight sensors, for example. Invarious embodiments, the surgical visualization system 200 can determine(e.g. triangulate) the tissue-to-ureter distance d_(A) (or depth) basedon other distances and/or the surface mapping logic described herein.

As mentioned above, a surgical visualization system includes a controlsystem configured to control various aspects of the surgicalvisualization system. The control system can have a variety ofconfigurations. FIG. 9 illustrates one embodiment of a control system600 for a surgical visualization system, such as the surgicalvisualization system 100 of FIG. 1 , the surgical visualization system200 of FIG. 8 , or other surgical visualization system described herein.The control system 600 is a conversion system that integrates spectralsignature tissue identification and structured light tissue positioningto identify a critical structure, especially when those structure(s) areobscured by tissue, e.g., by fat, connective tissue, blood tissue,and/or organ(s), and/or by blood, and/or to detect tissue variability,such as differentiating tumors and/or non-healthy tissue from healthytissue within an organ.

The control system 600 is configured for implementing a hyperspectralimaging and visualization system in which a molecular response isutilized to detect and identify anatomy in a surgical field of view. Thecontrol system 600 includes a conversion logic circuit 648 configured toconvert tissue data to usable information for surgeons and/or othermedical practitioners. For example, variable reflectance based onwavelengths with respect to obscuring material can be utilized toidentify the critical structure in the anatomy. Moreover, the controlsystem 600 is configured to combine the identified spectral signatureand the structural light data in an image. For example, the controlsystem 600 can be employed to create of three-dimensional data set forsurgical use in a system with augmentation image overlays. Techniquescan be employed both intraoperatively and preoperatively usingadditional visual information. In various embodiments, the controlsystem 600 is configured to provide warnings to a medical practitionerwhen in the proximity of one or more critical structures. Variousalgorithms can be employed to guide robotic automation andsemi-automated approaches based on the surgical procedure and proximityto the critical structure(s).

A projected array of lights is employed by the control system 600 todetermine tissue shape and motion intraoperatively. Alternatively, flashLidar may be utilized for surface mapping of the tissue.

The control system 600 is configured to detect the critical structure,which as mentioned above can include one or more critical structures,and provide an image overlay of the critical structure and measure thedistance to the surface of the visible tissue and the distance to theembedded/buried critical structure(s). The control system 600 canmeasure the distance to the surface of the visible tissue or detect thecritical structure and provide an image overlay of the criticalstructure.

The control system 600 includes a spectral control circuit 602. Thespectral control circuit 602 can be a field programmable gate array(FPGA) or another suitable circuit configuration, such as theconfigurations described with respect to FIG. 6 , FIG. 7 , and FIG. 8 .The spectral control circuit 602 includes a processor 604 configured toreceive video input signals from a video input processor 606. Theprocessor 604 can be configured for hyperspectral processing and canutilize C/C++ code, for example. The video input processor 606 isconfigured to receive video-in of control (metadata) data such asshutter time, wave length, and sensor analytics, for example. Theprocessor 604 is configured to process the video input signal from thevideo input processor 606 and provide a video output signal to a videooutput processor 608, which includes a hyperspectral video-out ofinterface control (metadata) data, for example. The video outputprocessor 608 is configured to provides the video output signal to animage overlay controller 610.

The video input processor 606 is operatively coupled to a camera 612 atthe patient side via a patient isolation circuit 614. The camera 612includes a solid state image sensor 634. The patient isolation circuit614 can include a plurality of transformers so that the patient isisolated from other circuits in the system. The camera 612 is configuredto receive intraoperative images through optics 632 and the image sensor634. The image sensor 634 can include a CMOS image sensor, for example,or can include another image sensor technology, such as those discussedherein in connection with FIG. 4 . The camera 612 is configured tooutput 613 images in 14 bit/pixel signals. A person skilled in the artwill appreciate that higher or lower pixel resolutions can be employed.The isolated camera output signal 613 is provided to a color RGB fusioncircuit 616, which in this illustrated embodiment employs a hardwareregister 618 and a Nios2 co-processor 620 configured to process thecamera output signal 613. A color RGB fusion output signal is providedto the video input processor 606 and a laser pulsing control circuit622.

The laser pulsing control circuit 622 is configured to control a laserlight engine 624. The laser light engine 624 is configured to outputlight in a plurality of wavelengths (λ1, μ2, μ3 . . . λn) including nearinfrared (NIR). The laser light engine 624 can operate in a plurality ofmodes. For example, the laser light engine 624 can operate in two modes.In a first mode, e.g., a normal operating mode, the laser light engine624 is configured to output an illuminating signal. In a second mode,e.g., an identification mode, the laser light engine 624 is configuredto output RGBG and NIR light. In various embodiments, the laser lightengine 624 can operate in a polarizing mode.

Light output 626 from the laser light engine 624 is configured toilluminate targeted anatomy in an intraoperative surgical site 627. Thelaser pulsing control circuit 622 is also configured to control a laserpulse controller 628 for a laser pattern projector 630 configured toproject a laser light pattern 631, such as a grid or pattern of linesand/or dots, at a predetermined wavelength (λ2) on an operative tissueor organ at the surgical site 627. The camera 612 is configured toreceive the patterned light as well as the reflected light outputthrough the camera optics 632. The image sensor 634 is configured toconvert the received light into a digital signal.

The color RGB fusion circuit 616 is also configured to output signals tothe image overlay controller 610 and a video input module 636 forreading the laser light pattern 631 projected onto the targeted anatomyat the surgical site 627 by the laser pattern projector 630. Aprocessing module 638 is configured to process the laser light pattern631 and output a first video output signal 640 representative of thedistance to the visible tissue at the surgical site 627. The data isprovided to the image overlay controller 610. The processing module 638is also configured to output a second video signal 642 representative ofa three-dimensional rendered shape of the tissue or organ of thetargeted anatomy at the surgical site.

The first and second video output signals 640, 642 include datarepresentative of the position of the critical structure on athree-dimensional surface model, which is provided to an integrationmodule 643. In combination with data from the video out processor 608 ofthe spectral control circuit 602, the integration module 643 isconfigured to determine the distance (e.g., distance d_(A) of FIG. 1 )to a buried critical structure (e.g., via triangularization algorithms644), and the distance to the buried critical structure can be providedto the image overlay controller 610 via a video out processor 646. Theforegoing conversion logic can encompass the conversion logic circuit648 intermediate video monitors 652 and the camera 624/laser patternprojector 630 positioned at the surgical site 627.

Preoperative data 650, such as from a CT or MRI scan, can be employed toregister or align certain three-dimensional deformable tissue in variousinstances. Such preoperative data 650 can be provided to the integrationmodule 643 and ultimately to the image overlay controller 610 so thatsuch information can be overlaid with the views from the camera 612 andprovided to the video monitors 652. Embodiments of registration ofpreoperative data are further described in U.S. Pat. Pub. No.2020/0015907 entitled “Integration Of Imaging Data” filed Sep. 11, 2018,which is hereby incorporated by reference herein in its entirety.

The video monitors 652 are configured to output the integrated/augmentedviews from the image overlay controller 610. A medical practitioner canselect and/or toggle between different views on one or more displays. Ona first display 652 a, which is a monitor in this illustratedembodiment, the medical practitioner can toggle between (A) a view inwhich a three-dimensional rendering of the visible tissue is depictedand (B) an augmented view in which one or more hidden criticalstructures are depicted over the three-dimensional rendering of thevisible tissue. On a second display 652 b, which is a monitor in thisillustrated embodiment, the medical practitioner can toggle on distancemeasurements to one or more hidden critical structures and/or thesurface of visible tissue, for example.

The various surgical visualization systems described herein can beutilized to visualize various different types of tissues and/oranatomical structures, including tissues and/or anatomical structuresthat may be obscured from being visualized by EMR in the visible portionof the spectrum. The surgical visualization system can utilize aspectral imaging system, as mentioned above, which can be configured tovisualize different types of tissues based upon their varyingcombinations of constituent materials. In particular, a spectral imagingsystem can be configured to detect the presence of various constituentmaterials within a tissue being visualized based on the absorptioncoefficient of the tissue across various EMR wavelengths. The spectralimaging system can be configured to characterize the tissue type of thetissue being visualized based upon the particular combination ofconstituent materials.

FIG. 10 shows a graph 300 depicting how the absorption coefficient ofvarious biological materials varies across the EMR wavelength spectrum.In the graph 300, the vertical axis 302 represents absorptioncoefficient of the biological material in cm⁻¹, and the horizontal axis304 represents EMR wavelength in μm. A first line 306 in the graph 300represents the absorption coefficient of water at various EMRwavelengths, a second line 308 represents the absorption coefficient ofprotein at various EMR wavelengths, a third line 310 represents theabsorption coefficient of melanin at various EMR wavelengths, a fourthline 312 represents the absorption coefficient of deoxygenatedhemoglobin at various EMR wavelengths, a fifth line 314 represents theabsorption coefficient of oxygenated hemoglobin at various EMRwavelengths, and a sixth line 316 represents the absorption coefficientof collagen at various EMR wavelengths. Different tissue types havedifferent combinations of constituent materials and, therefore, thetissue type(s) being visualized by a surgical visualization system canbe identified and differentiated between according to the particularcombination of detected constituent materials. Accordingly, a spectralimaging system of a surgical visualization system can be configured toemit EMR at a number of different wavelengths, determine the constituentmaterials of the tissue based on the detected absorption EMR absorptionresponse at the different wavelengths, and then characterize the tissuetype based on the particular detected combination of constituentmaterials.

FIG. 11 shows an embodiment of the utilization of spectral imagingtechniques to visualize different tissue types and/or anatomicalstructures. In FIG. 11 , a spectral emitter 320 (e.g., the spectrallight source 150 of FIG. 4 ) is being utilized by an imaging system tovisualize a surgical site 322. The EMR emitted by the spectral emitter320 and reflected from the tissues and/or structures at the surgicalsite 322 is received by an image sensor (e.g., the image sensor 135 ofFIG. 4 ) to visualize the tissues and/or structures, which can be eithervisible (e.g., be located at a surface of the surgical site 322) orobscured (e.g., underlay other tissue and/or structures at the surgicalsite 322). In this embodiment, an imaging system (e.g., the imagingsystem 142 of FIG. 4 ) visualizes a tumor 324, an artery 326, andvarious abnormalities 328 (e.g., tissues not confirming to known orexpected spectral signatures) based upon the spectral signaturescharacterized by the differing absorptive characteristics (e.g.,absorption coefficient) of the constituent materials for each of thedifferent tissue/structure types. The visualized tissues and structurescan be displayed on a display screen associated with or coupled to theimaging system (e.g., the display 146 of the imaging system 142 of FIG.4 ), on a primary display (e.g., the primary display 819 of FIG. 19 ),on a non-sterile display (e.g., the non-sterile displays 807, 809 ofFIG. 19 ), on a display of a surgical hub (e.g., the display of thesurgical hub 806 of FIG. 19 ), on a device/instrument display, and/or onanother display.

The imaging system can be configured to tailor or update the displayedsurgical site visualization according to the identified tissue and/orstructure types. For example, as shown in FIG. 11 , the imaging systemcan display a margin 330 associated with the tumor 324 being visualizedon a display screen associated with or coupled to the imaging system, ona primary display, on a non-sterile display, on a display of a surgicalhub, on a device/instrument display, and/or on another display. Themargin 330 can indicate the area or amount of tissue that should beexcised to ensure complete removal of the tumor 324. The surgicalvisualization system's control system (e.g., the control system 133 ofFIG. 4 ) can be configured to control or update the dimensions of themargin 330 based on the tissues and/or structures identified by theimaging system. In this illustrated embodiment, the imaging system hasidentified multiple abnormalities 328 within the field of view (FOV).Accordingly, the control system can adjust the displayed margin 330 to afirst updated margin 332 having sufficient dimensions to encompass theabnormalities 328. Further, the imaging system has also identified theartery 326 partially overlapping with the initially displayed margin 330(as indicated by a highlighted region 334 of the artery 326).Accordingly, the control system can adjust the displayed margin to asecond updated margin 336 having sufficient dimensions to encompass therelevant portion of the artery 326.

Tissues and/or structures can also be imaged or characterized accordingto their reflective characteristics, in addition to or in lieu of theirabsorptive characteristics described above with respect to FIG. 10 andFIG. 11 , across the EMR wavelength spectrum. For example, FIG. 12 ,FIG. 13 , and FIG. 14 illustrate various graphs of reflectance ofdifferent types of tissues or structures across different EMRwavelengths. FIG. 12 is a graphical representation 340 of anillustrative ureter signature versus obscurants. FIG. 13 is a graphicalrepresentation 342 of an illustrative artery signature versusobscurants. FIG. 14 is a graphical representation 344 of an illustrativenerve signature versus obscurants. The plots in FIG. 12 , FIG. 13 , andFIG. 14 represent reflectance as a function of wavelength (nm) for theparticular structures (ureter, artery, and nerve) relative to thecorresponding reflectances of fat, lung tissue, and blood at thecorresponding wavelengths. These graphs are simply for illustrativepurposes and it should be understood that other tissues and/orstructures could have corresponding detectable reflectance signaturesthat would allow the tissues and/or structures to be identified andvisualized.

Select wavelengths for spectral imaging can be identified and utilizedbased on the anticipated critical structures and/or obscurants at asurgical site (e.g., “selective spectral” imaging). By utilizingselective spectral imaging, the amount of time required to obtain thespectral image can be minimized such that the information can beobtained in real-time and utilized intraoperatively. The wavelengths canbe selected by a medical practitioner or by a control circuit based oninput by a user, e.g., a medical practitioner. In certain instances, thewavelengths can be selected based on machine learning and/or big dataaccessible to the control circuit via, e.g., a cloud or surgical hub.

FIG. 15 illustrates one embodiment of spectral imaging to tissue beingutilized intraoperatively to measure a distance between a waveformemitter and a critical structure that is obscured by tissue. FIG. 15shows an embodiment of a time-of-flight sensor system 404 utilizingwaveforms 424, 425. The time-of-flight sensor system 404 can beincorporated into a surgical visualization system, e.g., as the sensorsystem 104 of the surgical visualization system 100 of FIG. 1 . Thetime-of-flight sensor system 404 includes a waveform emitter 406 and awaveform receiver 408 on the same surgical device 402 (e.g., the emitter106 and the receiver 108 on the same surgical device 102 of FIG. 1 ).The emitted wave 400 extends to a critical structure 401 (e.g., thecritical structure 101 of FIG. 1 ) from the emitter 406, and thereceived wave 425 is reflected back to by the receiver 408 from thecritical structure 401. The surgical device 402 in this illustratedembodiment is positioned through a trocar 410 that extends into a cavity407 in a patient. Although the trocar 410 is used in this in thisillustrated embodiment, other trocars or other access devices can beused, or no access device may be used.

The waveforms 424, 425 are configured to penetrate obscuring tissue 403,such as by having wavelengths in the NIR or SWIR spectrum ofwavelengths. A spectral signal (e.g., hyperspectral, multispectral, orselective spectral) or a photoacoustic signal is emitted from theemitter 406, as shown by a first arrow 407 pointing distally, and canpenetrate the tissue 403 in which the critical structure 401 isconcealed. The emitted waveform 424 is reflected by the criticalstructure 401, as shown by a second arrow 409 pointing proximally. Thereceived waveform 425 can be delayed due to a distance d between adistal end of the surgical device 402 and the critical structure 401.The waveforms 424, 425 can be selected to target the critical structure401 within the tissue 403 based on the spectral signature of thecritical structure 401, as described herein. The emitter 406 isconfigured to provide a binary signal on and off, as shown in FIG. 16 ,for example, which can be measured by the receiver 408.

Based on the delay between the emitted wave 424 and the received wave425, the time-of-flight sensor system 404 is configured to determine thedistance d. A time-of-flight timing diagram 430 for the emitter 406 andthe receiver 408 of FIG. 15 is shown in FIG. 16 . The delay is afunction of the distance d and the distance d is given by:

$d = {\frac{ct}{2} \cdot \frac{q_{2}}{q_{1} + q_{2}}}$

where c=the speed of light; t=length of pulse; q1=accumulated chargewhile light is emitted; and q2=accumulated charge while light is notbeing emitted.

The time-of-flight of the waveforms 424, 425 corresponds to the distanced in FIG. 15 . In various instances, additional emitters/receiversand/or pulsing signals from the emitter 406 can be configured to emit anon-penetrating signal. The non-penetrating signal can be configured todetermine the distance from the emitter 406 to the surface 405 of theobscuring tissue 403. In various instances, a depth of the criticalstructure 401 can be determined by:

d _(A) =d _(w) −d _(t)

where d_(A)=the depth of the critical structure 401; d_(w)=the distancefrom the emitter 406 to the critical structure 401 (d in FIG. 15 ); andd_(t,)=the distance from the emitter 406 (on the distal end of thesurgical device 402) to the surface 405 of the obscuring tissue 403.

FIG. 17 illustrates another embodiment of a time-of-flight sensor system504 utilizing waves 524 a, 524 b, 524 c, 525 a, 525 b, 525 c is shown.The time-of-flight sensor system 504 can be incorporated into a surgicalvisualization system, e.g., as the sensor system 104 of the surgicalvisualization system 100 of FIG. 1 . The time-of-flight sensor system504 includes a waveform emitter 506 and a waveform receiver 508 (e.g.,the emitter 106 and the receiver 108 of FIG. 1 ). The waveform emitter506 is positioned on a first surgical device 502 a (e.g., the surgicaldevice 102 of FIG. 1 ), and the waveform receiver 508 is positioned on asecond surgical device 502 b. The surgical devices 502 a, 502 b arepositioned through first and second trocars 510 a, 510 b, respectively,which extend into a cavity 507 in a patient. Although the trocars 510 a,510 b are used in this in this illustrated embodiment, other trocars orother access devices can be used, or no access device may be used. Theemitted waves 524 a, 524 b, 524 c extend toward a surgical site from theemitter 506, and the received waves 525 a, 525 b, 525 c are reflectedback to the receiver 508 from various structures and/or surfaces at thesurgical site.

The different emitted waves 524 a, 524 b, 524 c are configured to targetdifferent types of material at the surgical site. For example, the wave524 a targets obscuring tissue 503, the wave 524 b targets a firstcritical structure 501 a (e.g., the critical structure 101 of FIG. 1 ),which is a vessel in this illustrated embodiment, and the wave 524 ctargets a second critical structure 501 b (e.g., the critical structure101 of FIG. 1 ), which is a cancerous tumor in this illustratedembodiment. The wavelengths of the waves 524 a, 524 b, 524 c can be inthe visible light, NIR, or SWIR spectrum of wavelengths. For example,visible light can be reflected off a surface 505 of the tissue 503, andNIR and/or SWIR waveforms can penetrate the surface 505 of the tissue503. In various aspects, as described herein, a spectral signal (e.g.,hyperspectral, multispectral, or selective spectral) or a photoacousticsignal can be emitted from the emitter 506. The waves 524 b, 524 c canbe selected to target the critical structures 501 a, 501 b within thetissue 503 based on the spectral signature of the critical structure 501a, 501 b, as described herein. Photoacoustic imaging is furtherdescribed in various U.S. patent applications, which are incorporated byreference herein in the present disclosure.

The emitted waves 524 a, 524 b, 524 c are reflected off the targetedmaterial, namely the surface 505, the first critical structure 501 a,and the second structure 501 b, respectively. The received waveforms 525a, 525 b, 525 c can be delayed due to distances d_(1a), d_(2a), d_(3a),d_(1b), d_(2b), d_(2c).

In the time-of-flight sensor system 504, in which the emitter 506 andthe receiver 508 are independently positionable (e.g., on separatesurgical devices 502 a, 502 b and/or controlled by separate roboticarms), the various distances d_(1a), d_(2a), d_(3a), d_(1b), d_(2b),d_(2c) can be calculated from the known position of the emitter 506 andthe receiver 508. For example, the positions can be known when thesurgical devices 502 a, 502 b are robotically-controlled. Knowledge ofthe positions of the emitter 506 and the receiver 508, as well as thetime of the photon stream to target a certain tissue and the informationreceived by the receiver 508 of that particular response can allow adetermination of the distances d_(1a), d_(2a), d_(3a), d_(1b), d_(2b),d_(2c). In one aspect, the distance to the obscured critical structures501 a, 501 b can be triangulated using penetrating wavelengths. Becausethe speed of light is constant for any wavelength of visible orinvisible light, the time-of-flight sensor system 504 can determine thevarious distances.

In a view provided to the medical practitioner, such as on a display,the receiver 508 can be rotated such that a center of mass of the targetstructure in the resulting images remains constant, e.g., in a planeperpendicular to an axis of a select target structure 503, 501 a, or 501b. Such an orientation can quickly communicate one or more relevantdistances and/or perspectives with respect to the target structure. Forexample, as shown in FIG. 17 , the surgical site is displayed from aviewpoint in which the critical structure 501 a is perpendicular to theviewing plane (e.g., the vessel is oriented in/out of the page). Such anorientation can be default setting; however, the view can be rotated orotherwise adjusted by a medical practitioner. In certain instances, themedical practitioner can toggle between different surfaces and/or targetstructures that define the viewpoint of the surgical site provided bythe imaging system.

As in this illustrated embodiment, the receiver 508 can be mounted onthe trocar 510 b (or other access device) through which the surgicaldevice 502 b is positioned. In other embodiments, the receiver 508 canbe mounted on a separate robotic arm for which the three-dimensionalposition is known. In various instances, the receiver 508 can be mountedon a movable arm that is separate from a robotic surgical system thatcontrols the surgical device 502 a or can be mounted to an operatingroom (OR) table or fixture that is intraoperatively registerable to therobot coordinate plane. In such instances, the position of the emitter506 and the receiver 508 can be registerable to the same coordinateplane such that the distances can be triangulated from outputs from thetime-of-flight sensor system 504.

Combining time-of-flight sensor systems and near-infrared spectroscopy(NIRS), termed TOF-NIRS, which is capable of measuring the time-resolvedprofiles of NIR light with nanosecond resolution can be found in“Time-Of-Flight Near-Infrared Spectroscopy For NondestructiveMeasurement Of Internal Quality In Grapefruit,” Journal of the AmericanSociety for Horticultural Science, May 2013 vol. 138 no. 3 225-228,which is hereby incorporated by reference in its entirety.

Embodiments of visualization systems and aspects and uses thereof aredescribed further in U.S. Pat. Pub. No. 2020/0015923 entitled “SurgicalVisualization Platform” filed Sep. 11, 2018, U.S. Pat. Pub. No.2020/0015900 entitled “Controlling An Emitter Assembly Pulse Sequence”filed Sep. 11, 2018, U.S. Pat. Pub. No. 2020/0015668 entitled “SingularEMR Source Emitter Assembly” filed Sep. 11, 2018, U.S. Pat. Pub. No.2020/0015925 entitled “Combination Emitter And Camera Assembly” filedSep. 11, 2018, U.S. Pat. Pub. No. 2020/00015899 entitled “SurgicalVisualization With Proximity Tracking Features” filed Sep. 11, 2018,U.S. Pat. Pub. No. 2020/00015903 entitled “Surgical Visualization OfMultiple Targets” filed Sep. 11, 2018, U.S. Pat. No. 10,792,034 entitled“Visualization Of Surgical Devices” filed Sep. 11, 2018, U.S. Pat. Pub.No. 2020/0015897 entitled “Operative Communication Of Light” filed Sep.11, 2018, U.S. Pat. Pub. No. 2020/0015924 entitled “Robotic LightProjection Tools” filed Sep. 11, 2018, U.S. Pat. Pub. No. 2020/0015898entitled “Surgical Visualization Feedback System” filed Sep. 11, 2018,U.S. Pat. Pub. No. 2020/0015906 entitled “Surgical Visualization AndMonitoring” filed Sep. 11, 2018, U.S. Pat. Pub. No. 2020/0015907entitled “Integration Of Imaging Data” filed Sep. 11, 2018, U.S. Pat.No. 10,925,598 entitled “Robotically-Assisted Surgical Suturing Systems”filed Sep. 11, 2018, U.S. Pat. Pub. No. 2020/0015901 entitled “SafetyLogic For Surgical Suturing Systems” filed Sep. 11, 2018, U.S. Pat. Pub.No. 2020/0015914 entitled “Robotic Systems With Separate PhotoacousticReceivers” filed Sep. 11, 2018, U.S. Pat. Pub. No. 2020/0015902 entitled“Force Sensor Through Structured Light Deflection” filed Sep. 11, 2018,U.S. Pat. Pub. No. 2019/0201136 entitled “Method Of Hub Communication”filed Dec. 4, 2018, U.S. patent application Ser. No. 16/729,772 entitled“Analyzing Surgical Trends By A Surgical System” filed Dec. 30, 2019,U.S. patent application Ser. No. 16/729,747 entitled “Dynamic SurgicalVisualization Systems” filed Dec. 30, 2019, U.S. patent application Ser.No. 16/729,744 entitled “Visualization Systems Using Structured Light”filed Dec. 30, 2019, U.S. patent application Ser. No. 16/729,778entitled “System And Method For Determining, Adjusting, And ManagingResection Margin About A Subject Tissue” filed Dec. 30, 2019, U.S.patent application Ser. No. 16/729,729 entitled “Surgical Systems ForProposing And Corroborating Organ Portion Removals” filed Dec. 30, 2019,U.S. patent application Ser. No. 16/729,778 entitled “Surgical SystemFor Overlaying Surgical Instrument Data Onto A Virtual Three DimensionalConstruct Of An Organ” filed Dec. 30, 2019, U.S. patent application Ser.No. 16/729,751 entitled “Surgical Systems For Generating ThreeDimensional Constructs Of Anatomical Organs And Coupling IdentifiedAnatomical Structures Thereto” filed Dec. 30, 2019, U.S. patentapplication Ser. No. 16/729,740 entitled “Surgical Systems CorrelatingVisualization Data And Powered Surgical Instrument Data” filed Dec. 30,2019, U.S. patent application Ser. No. 16/729,737 entitled “AdaptiveSurgical System Control According To Surgical Smoke CloudCharacteristics” filed Dec. 30, 2019, U.S. patent application Ser. No.16/729,796 entitled “Adaptive Surgical System Control According ToSurgical Smoke Particulate Characteristics” filed Dec. 30, 2019, U.S.patent application Ser. No. 16/729,803 entitled “Adaptive VisualizationBy A Surgical System” filed Dec. 30, 2019, U.S. patent application Ser.No. 16/729,807 entitled “Method Of Using Imaging Devices In Surgery”filed Dec. 30, 2019, U.S. Pat. App No. 63/249,644 entitled “SurgicalDevices, Systems, And Methods Using Multi-Source Imaging” filed on Sep.29, 2021, U.S. Pat. App No. 63/249,652 entitled “Surgical Devices,Systems, Methods Using Fiducial Identification And Tracking” filed onSep. 29, 2021, U.S. Pat. App No. 63/249,658 entitled “Surgical Devices,Systems, And Methods For Control Of One Visualization With Another”filed on Sep. 29, 2021, U.S. Pat. App No. 63/249,870 entitled “MethodsAnd Systems For Controlling Cooperative Surgical Instruments” filed onSep. 29, 2021, U.S. Pat. App No. 63/249,877 entitled “Methods AndSystems For Controlling Cooperative Surgical Instruments” filed on Sep.29, 2021, and U.S. Pat. App No. 63/249,980 entitled “Cooperative Access”filed on Sep. 29, 2021, which are hereby incorporated by reference intheir entireties.

Surgical Hubs

The various visualization or imaging systems described herein can beincorporated into a system that includes a surgical hub. In general, asurgical hub can be a component of a comprehensive digital medicalsystem capable of spanning multiple medical facilities and configured toprovide integrated and comprehensive improved medical care to a vastnumber of patients. The comprehensive digital medical system includes acloud-based medical analytics system that is configured to interconnectto multiple surgical hubs located across many different medicalfacilities. The surgical hubs are configured to interconnect with one ormore elements, such as one or more surgical instruments that are used toconduct medical procedures on patients and/or one or more visualizationsystems that are used during performance of medical procedures. Thesurgical hubs provide a wide array of functionality to improve theoutcomes of medical procedures. The data generated by the varioussurgical devices, visualization systems, and surgical hubs about thepatient and the medical procedure may be transmitted to the cloud-basedmedical analytics system. This data may then be aggregated with similardata gathered from many other surgical hubs, visualization systems, andsurgical instruments located at other medical facilities. Variouspatterns and correlations may be found through the cloud-based analyticssystem analyzing the collected data. Improvements in the techniques usedto generate the data may be generated as a result, and theseimprovements may then be disseminated to the various surgical hubs,visualization systems, and surgical instruments. Due to theinterconnectedness of all of the aforementioned components, improvementsin medical procedures and practices may be found that otherwise may notbe found if the many components were not so interconnected.

Examples of surgical hubs configured to receive, analyze, and outputdata, and methods of using such surgical hubs, are further described inU.S. Pat. Pub. No. 2019/0200844 entitled “Method Of Hub Communication,Processing, Storage And Display” filed Dec. 4, 2018, U.S. Pat. Pub. No.2019/0200981 entitled “Method Of Compressing Tissue Within A StaplingDevice And Simultaneously Displaying The Location Of The Tissue WithinThe Jaws” filed Dec. 4, 2018, U.S. Pat. Pub. No. 2019/0201046 entitled“Method For Controlling Smart Energy Devices” filed Dec. 4, 2018, U.S.Pat. Pub. No. 2019/0201114 entitled “Adaptive Control Program UpdatesFor Surgical Hubs” filed Mar. 29, 2018, U.S. Pat. Pub. No. 2019/0201140entitled “Surgical Hub Situational Awareness” filed Mar. 29, 2018, U.S.Pat. Pub. No. 2019/0206004 entitled “Interactive Surgical Systems WithCondition Handling Of Devices And Data Capabilities” filed Mar. 29,2018, U.S. Pat. Pub. No. 2019/0206555 entitled “Cloud-based MedicalAnalytics For Customization And Recommendations To A User” filed Mar.29, 2018, and U.S. Pat. Pub. No. 2019/0207857 entitled “Surgical NetworkDetermination Of Prioritization Of Communication, Interaction, OrProcessing Based On System Or Device Needs” filed Nov. 6, 2018, whichare hereby incorporated by reference in their entireties.

FIG. 18 illustrates one embodiment of a computer-implemented interactivesurgical system 700 that includes one or more surgical systems 702 and acloud-based system (e.g., a cloud 704 that can include a remote server713 coupled to a storage device 705). Each surgical system 702 includesat least one surgical hub 706 in communication with the cloud 704. Inone example, as illustrated in FIG. 18 , the surgical system 702includes a visualization system 708, a robotic system 710, and anintelligent (or “smart”) surgical instrument 712, which are configuredto communicate with one another and/or the hub 706. The intelligentsurgical instrument 712 can include imaging device(s). The surgicalsystem 702 can include an M number of hubs 706, an N number ofvisualization systems 708, an O number of robotic systems 710, and a Pnumber of intelligent surgical instruments 712, where M, N, 0, and P areintegers greater than or equal to one that may or may not be equal toany one or more of each other. Various exemplary intelligent surgicalinstruments and robotic systems are described herein.

Data received by a surgical hub from a surgical visualization system canbe used in any of a variety of ways. In an exemplary embodiment, thesurgical hub can receive data from a surgical visualization system inuse with a patient in a surgical setting, e.g., in use in an operatingroom during performance of a surgical procedure. The surgical hub canuse the received data in any of one or more ways, as discussed herein.

The surgical hub can be configured to analyze received data in real timewith use of the surgical visualization system and adjust control one ormore of the surgical visualization system and/or one or more intelligentsurgical instruments in use with the patient based on the analysis ofthe received data. Such adjustment can include, for example, adjustingone or operational control parameters of intelligent surgicalinstrument(s), causing one or more sensors of one or more intelligentsurgical instruments to take a measurement to help gain an understandingof the patient's current physiological condition, and/or currentoperational status of an intelligent surgical instrument, and otheradjustments. Controlling and adjusting operation of intelligent surgicalinstruments is discussed further below. Examples of operational controlparameters of an intelligent surgical instrument include motor speed,cutting element speed, time, duration, level of energy application, andlight emission. Examples of surgical hubs and of controlling andadjusting intelligent surgical instrument operation are describedfurther in previously mentioned U.S. patent application Ser. No.16/729,772 entitled “Analyzing Surgical Trends By A Surgical System”filed Dec. 30, 2019, U.S. patent application Ser. No. 16/729,747entitled “Dynamic Surgical Visualization Systems” filed Dec. 30, 2019,U.S. patent application Ser. No. 16/729,744 entitled “VisualizationSystems Using Structured Light” filed Dec. 30, 2019, U.S. patentapplication Ser. No. 16/729,778 entitled “System And Method ForDetermining, Adjusting, And Managing Resection Margin About A SubjectTissue” filed Dec. 30, 2019, U.S. patent application Ser. No. 16/729,729entitled “Surgical Systems For Proposing And Corroborating Organ PortionRemovals” filed Dec. 30, 2019, U.S. patent application Ser. No.16/729,778 entitled “Surgical System For Overlaying Surgical InstrumentData Onto A Virtual Three Dimensional Construct Of An Organ” filed Dec.30, 2019, U.S. patent application Ser. No. 16/729,751 entitled “SurgicalSystems For Generating Three Dimensional Constructs Of Anatomical OrgansAnd Coupling Identified Anatomical Structures Thereto” filed Dec. 30,2019, U.S. patent application Ser. No. 16/729,740 entitled “SurgicalSystems Correlating Visualization Data And Powered Surgical InstrumentData” filed Dec. 30, 2019, U.S. patent application Ser. No. 16/729,737entitled “Adaptive Surgical System Control According To Surgical SmokeCloud Characteristics” filed Dec. 30, 2019, U.S. patent application Ser.No. 16/729,796 entitled “Adaptive Surgical System Control According ToSurgical Smoke Particulate Characteristics” filed Dec. 30, 2019, U.S.patent application Ser. No. 16/729,803 entitled “Adaptive VisualizationBy A Surgical System” filed Dec. 30, 2019, and U.S. patent applicationSer. No. 16/729,807 entitled “Method Of Using Imaging Devices InSurgery” filed Dec. 30, 2019, and in U.S. patent application Ser. No.17/068,857 entitled “Adaptive Responses From Smart Packaging Of DrugDelivery Absorbable Adjuncts” filed Oct. 13, 2020, U.S. patentapplication Ser. No. 17/068,858 entitled “Drug Administration DevicesThat Communicate With Surgical Hubs” filed Oct. 13, 2020, U.S. patentapplication Ser. No. 17/068,859 entitled “Controlling Operation Of DrugAdministration Devices Using Surgical Hubs” filed Oct. 13, 2020, U.S.patent application Ser. No. 17/068,863 entitled “Patient MonitoringUsing Drug Administration Devices” filed Oct. 13, 2020, U.S. patentapplication Ser. No. 17/068,865 entitled “Monitoring And CommunicatingInformation Using Drug Administration Devices” filed Oct. 13, 2020, andU.S. patent application Ser. No. 17/068,867 entitled “Aggregating AndAnalyzing Drug Administration Data” filed Oct. 13, 2020, which arehereby incorporated by reference in their entireties.

The surgical hub can be configured to cause visualization of thereceived data to be provided in the surgical setting on a display sothat a medical practitioner in the surgical setting can view the dataand thereby receive an understanding of the operation of the imagingdevice(s) in use in the surgical setting. Such information provided viavisualization can include text and/or images.

FIG. 19 illustrates one embodiment of a surgical system 802 including asurgical hub 806 (e.g., the surgical hub 706 of FIG. 18 or othersurgical hub described herein), a robotic surgical system 810 (e.g., therobotic surgical system 110 of FIG. 1 or other robotic surgical systemherein), and a visualization system 808 (e.g., the visualization system100 of FIG. 1 or other visualization system described herein). Thesurgical hub 806 can be in communication with a cloud, as discussedherein. FIG. 19 shows the surgical system 802 being used to perform asurgical procedure on a patient who is lying down on an operating table814 in a surgical operating room 816. The robotic system 810 includes asurgeon's console 818, a patient side cart 820 (surgical robot), and arobotic system surgical hub 822. The robotic system surgical hub 822 isgenerally configured similar to the surgical hub 822 and can be incommunication with a cloud. In some embodiments, the robotic systemsurgical hub 822 and the surgical hub 806 can be combined. The patientside cart 820 can manipulate an intelligent surgical tool 812 through aminimally invasive incision in the body of the patient while a medicalpractitioner, e.g., a surgeon, nurse, and/or other medical practitioner,views the surgical site through the surgeon's console 818. An image ofthe surgical site can be obtained by an imaging device 824 (e.g., theimaging device 120 of FIG. 1 or other imaging device described herein),which can be manipulated by the patient side cart 820 to orient theimaging device 824. The robotic system surgical hub 822 can be used toprocess the images of the surgical site for subsequent display to thesurgeon through the surgeon's console 818.

A primary display 819 is positioned in the sterile field of theoperating room 816 and is configured to be visible to an operator at theoperating table 814. In addition, as in this illustrated embodiment, avisualization tower 818 can positioned outside the sterile field. Thevisualization tower 818 includes a first non-sterile display 807 and asecond non-sterile display 809, which face away from each other. Thevisualization system 808, guided by the surgical hub 806, is configuredto utilize the displays 807, 809, 819 to coordinate information flow tomedical practitioners inside and outside the sterile field. For example,the surgical hub 806 can cause the visualization system 808 to display asnapshot and/or a video of a surgical site, as obtained by the imagingdevice 824, on one or both of the non-sterile displays 807, 809, whilemaintaining a live feed of the surgical site on the primary display 819.The snapshot and/or video on the non-sterile display 807 and/or 809 canpermit a non-sterile medical practitioner to perform a diagnostic steprelevant to the surgical procedure, for example.

The surgical hub 806 is configured to route a diagnostic input orfeedback entered by a non-sterile medical practitioner at thevisualization tower 818 to the primary display 819 within the sterilefield, where it can be viewed by a sterile medical practitioner at theoperating table 814. For example, the input can be in the form of amodification to the snapshot and/or video displayed on the non-steriledisplay 807 and/or 809, which can be routed to the primary display 819by the surgical hub 806.

The surgical hub 806 is configured to coordinate information flow to adisplay of the intelligent surgical instrument 812, as is described invarious U.S. patent applications that are incorporated by referenceherein in the present disclosure. A diagnostic input or feedback enteredby a non-sterile operator at the visualization tower 818 can be routedby the surgical hub 806 to the display 819 within the sterile field,where it can be viewed by the operator of the surgical instrument 812and/or by other medical practitioner(s) in the sterile field.

The intelligent surgical instrument 812 and the imaging device 824,which is also an intelligent surgical tool, is being used with thepatient in the surgical procedure as part of the surgical system 802.Other intelligent surgical instruments 812 a that can be used in thesurgical procedure, e.g., that can be removably coupled to the patientside cart 820 and be in communication with the robotic surgical system810 and the surgical hub 806, are also shown in FIG. 19 as beingavailable. Non-intelligent (or “dumb”) surgical instruments 817, e.g.,scissors, trocars, cannulas, scalpels, etc., that cannot be incommunication with the robotic surgical system 810 and the surgical hub806 are also shown in FIG. 19 as being available for use.

Operating Intelligent Surgical Instruments

An intelligent surgical device can have an algorithm stored thereon,e.g., in a memory thereof, configured to be executable on board theintelligent surgical device, e.g., by a processor thereof, to controloperation of the intelligent surgical device. In some embodiments,instead of or in addition to being stored on the intelligent surgicaldevice, the algorithm can be stored on a surgical hub, e.g., in a memorythereof, that is configured to communicate with the intelligent surgicaldevice.

The algorithm is stored in the form of one or more sets of pluralitiesof data points defining and/or representing instructions, notifications,signals, etc. to control functions of the intelligent surgical device.In some embodiments, data gathered by the intelligent surgical devicecan be used by the intelligent surgical device, e.g., by a processor ofthe intelligent surgical device, to change at least one variableparameter of the algorithm. As discussed above, a surgical hub can be incommunication with an intelligent surgical device, so data gathered bythe intelligent surgical device can be communicated to the surgical huband/or data gathered by another device in communication with thesurgical hub can be communicated to the surgical hub, and data can becommunicated from the surgical hub to the intelligent surgical device.Thus, instead of or in addition to the intelligent surgical device beingconfigured to change a stored variable parameter, the surgical hub canbe configured to communicate the changed at least one variable, alone oras part of the algorithm, to the intelligent surgical device and/or thesurgical hub can communicate an instruction to the intelligent surgicaldevice to change the at least one variable as determined by the surgicalhub.

The at least one variable parameter is among the algorithm's datapoints, e.g., are included in instructions for operating the intelligentsurgical device, and are thus each able to be changed by changing one ormore of the stored pluralities of data points of the algorithm. Afterthe at least one variable parameter has been changed, subsequentexecution of the algorithm is according to the changed algorithm. Assuch, operation of the intelligent surgical device over time can bemanaged for a patient to increase the beneficial results use of theintelligent surgical device by taking into consideration actualsituations of the patient and actual conditions and/or results of thesurgical procedure in which the intelligent surgical device is beingused. Changing the at least one variable parameter is automated toimprove patient outcomes. Thus, the intelligent surgical device can beconfigured to provide personalized medicine based on the patient and thepatient's surrounding conditions to provide a smart system. In asurgical setting in which the intelligent surgical device is being usedduring performance of a surgical procedure, automated changing of the atleast one variable parameter may allow for the intelligent surgicaldevice to be controlled based on data gathered during the performance ofthe surgical procedure, which may help ensure that the intelligentsurgical device is used efficiently and correctly and/or may help reducechances of patient harm by harming a critical anatomical structure.

The at least one variable parameter can be any of a variety of differentoperational parameters. Examples of variable parameters include motorspeed, motor torque, energy level, energy application duration, tissuecompression rate, jaw closure rate, cutting element speed, loadthreshold, etc.

FIG. 20 illustrates one embodiment of an intelligent surgical instrument900 including a memory 902 having an algorithm 904 stored therein thatincludes at least one variable parameter. The algorithm 904 can be asingle algorithm or can include a plurality of algorithms, e.g.,separate algorithms for different aspects of the surgical instrument'soperation, where each algorithm includes at least one variableparameter. The intelligent surgical instrument 900 can be the surgicaldevice 102 of FIG. 1 , the imaging device 120 of FIG. 1 , the surgicaldevice 202 of FIG. 8 , the imaging device 220 of FIG. 8 , the surgicaldevice 402 of FIG. 15 , the surgical device 502 a of FIG. 17 , thesurgical device 502 b of FIG. 17 , the surgical device 712 of FIG. 18 ,the surgical device 812 of FIG. 19 , the imaging device 824 of FIG. 19 ,or other intelligent surgical instrument. The surgical instrument 900also includes a processor 906 configured to execute the algorithm 904 tocontrol operation of at least one aspect of the surgical instrument 900.To execute the algorithm 904, the processor 906 is configured to run aprogram stored in the memory 902 to access a plurality of data points ofthe algorithm 904 in the memory 902.

The surgical instrument 900 also includes a communications interface908, e.g., a wireless transceiver or other wired or wirelesscommunications interface, configured to communicate with another device,such as a surgical hub 910. The communications interface 908 can beconfigured to allow one-way communication, such as providing data to aremote server (e.g., a cloud server or other server) and/or to a local,surgical hub server, and/or receiving instructions or commands from aremote server and/or a local, surgical hub server, or two-waycommunication, such as providing information, messages, data, etc.regarding the surgical instrument 900 and/or data stored thereon andreceiving instructions, such as from a doctor; a remote server regardingupdates to software; a local, surgical hub server regarding updates tosoftware; etc.

The surgical instrument 900 is simplified in FIG. 20 and can includeadditional components, e.g., a bus system, a handle, a elongate shafthaving an end effector at a distal end thereof, a power source, etc. Theprocessor 906 can also be configured to execute instructions stored inthe memory 902 to control the device 900 generally, including otherelectrical components thereof such as the communications interface 908,an audio speaker, a user interface, etc.

The processor 906 is configured to change at least one variableparameter of the algorithm 904 such that a subsequent execution of thealgorithm 904 will be in accordance with the changed at least onevariable parameter. To change the at least one variable parameter of thealgorithm 904, the processor 906 is configured to modify or update thedata point(s) of the at least one variable parameter in the memory 902.The processor 906 can be configured to change the at least one variableparameter of the algorithm 904 in real time with use of the surgicaldevice 900 during performance of a surgical procedure, which mayaccommodate real time conditions.

Additionally or alternatively to the processor 906 changing the at leastone variable parameter, the processor 906 can be configured to changethe algorithm 904 and/or at least one variable parameter of thealgorithm 904 in response to an instruction received from the surgicalhub 910. In some embodiments, the processor 906 is configured to changethe at least one variable parameter only after communicating with thesurgical hub 910 and receiving an instruction therefrom, which may helpensure coordinated action of the surgical instrument 900 with otheraspects of the surgical procedure in which the surgical instrument 900is being used.

In an exemplary embodiment, the processor 906 executes the algorithm 904to control operation of the surgical instrument 900, changes the atleast one variable parameter of the algorithm 904 based on real timedata, and executes the algorithm 904 after changing the at least onevariable parameter to control operation of the surgical instrument 900.

FIG. 21 illustrates one embodiment of a method 912 of using of thesurgical instrument 900 including a change of at least one variableparameter of the algorithm 904. The processor 906 controls 914 operationof the surgical instrument 900 by executing the algorithm 904 stored inthe memory 902. Based on any of this subsequently known data and/orsubsequently gathered data, the processor 904 changes 916 the at leastone variable parameter of the algorithm 904 as discussed above. Afterchanging the at least one variable parameter, the processor 906 controls918 operation of the surgical instrument 900 by executing the algorithm904, now with the changed at least one variable parameter. The processor904 can change 916 the at least one variable parameter any number oftimes during performance of a surgical procedure, e.g., zero, one, two,three, etc. During any part of the method 912, the surgical instrument900 can communicate with one or more computer systems, e.g., thesurgical hub 910, a remote server such as a cloud server, etc., usingthe communications interface 908 to provide data thereto and/or receiveinstructions therefrom.

Situational Awareness

Operation of an intelligent surgical instrument can be altered based onsituational awareness of the patient. The operation of the intelligentsurgical instrument can be altered manually, such as by a user of theintelligent surgical instrument handling the instrument differently,providing a different input to the instrument, ceasing use of theinstrument, etc. Additionally or alternatively, the operation of anintelligent surgical instrument can be changed automatically by analgorithm of the instrument being changed, e.g., by changing at leastone variable parameter of the algorithm. As mentioned above, thealgorithm can be adjusted automatically without user input requestingthe change. Automating the adjustment during performance of a surgicalprocedure may help save time, may allow medical practitioners to focuson other aspects of the surgical procedure, and/or may ease the processof using the surgical instrument for a medical practitioner, which eachmay improve patient outcomes, such as by avoiding a critical structure,controlling the surgical instrument with consideration of a tissue typethe instrument is being used on and/or near, etc.

The visualization systems described herein can be utilized as part of asituational awareness system that can be embodied or executed by asurgical hub, e.g., the surgical hub 706, the surgical hub 806, or othersurgical hub described herein. In particular, characterizing,identifying, and/or visualizing surgical instruments (including theirpositions, orientations, and actions), tissues, structures, users,and/or other things located within the surgical field or the operatingtheater can provide contextual data that can be utilized by asituational awareness system to infer various information, such as atype of surgical procedure or a step thereof being performed, a type oftissue(s) and/or structure(s) being manipulated by a surgeon or othermedical practitioner, and other information. The contextual data canthen be utilized by the situational awareness system to provide alertsto a user, suggest subsequent steps or actions for the user toundertake, prepare surgical devices in anticipation for their use (e.g.,activate an electrosurgical generator in anticipation of anelectrosurgical instrument being utilized in a subsequent step of thesurgical procedure, etc.), control operation of intelligent surgicalinstruments (e.g., customize surgical instrument operational parametersof an algorithm as discussed further below), and so on.

Although an intelligent surgical device including an algorithm thatresponds to sensed data, e.g., by having at least one variable parameterof the algorithm changed, can be an improvement over a “dumb” devicethat operates without accounting for sensed data, some sensed data canbe incomplete or inconclusive when considered in isolation, e.g.,without the context of the type of surgical procedure being performed orthe type of tissue that is being operated on. Without knowing theprocedural context (e.g., knowing the type of tissue being operated onor the type of procedure being performed), the algorithm may control thesurgical device incorrectly or sub-optimally given the particularcontext-free sensed data. For example, the optimal manner for analgorithm to control a surgical instrument in response to a particularsensed parameter can vary according to the particular tissue type beingoperated on. This is due to the fact that different tissue types havedifferent properties (e.g., resistance to tearing, ease of being cut,etc.) and thus respond differently to actions taken by surgicalinstruments. Therefore, it may be desirable for a surgical instrument totake different actions even when the same measurement for a particularparameter is sensed. As one example, the optimal manner in which tocontrol a surgical stapler in response to the surgical stapler sensingan unexpectedly high force to close its end effector will vary dependingupon whether the tissue type is susceptible or resistant to tearing. Fortissues that are susceptible to tearing, such as lung tissue, thesurgical instrument's control algorithm would optimally ramp down themotor in response to an unexpectedly high force to close to avoidtearing the tissue, e.g., change a variable parameter controlling motorspeed or torque so the motor is slower. For tissues that are resistantto tearing, such as stomach tissue, the instrument's algorithm wouldoptimally ramp up the motor in response to an unexpectedly high force toclose to ensure that the end effector is clamped properly on the tissue,e.g., change a variable parameter controlling motor speed or torque sothe motor is faster. Without knowing whether lung or stomach tissue hasbeen clamped, the algorithm may be sub-optimally changed or not changedat all.

A surgical hub can be configured to derive information about a surgicalprocedure being performed based on data received from various datasources and then control modular devices accordingly. In other words,the surgical hub can be configured to infer information about thesurgical procedure from received data and then control the modulardevices operably coupled to the surgical hub based upon the inferredcontext of the surgical procedure. Modular devices can include anysurgical device that is controllable by a situational awareness system,such as visualization system devices (e.g., a camera, a display screen,etc.), smart surgical instruments (e.g., an ultrasonic surgicalinstrument, an electrosurgical instrument, a surgical stapler, smokeevacuators, scopes, etc.). A modular device can include sensor(s)sconfigured to detect parameters associated with a patient with which thedevice is being used and/or associated with the modular device itself.

The contextual information derived or inferred from the received datacan include, for example, a type of surgical procedure being performed,a particular step of the surgical procedure that the surgeon (or othermedical practitioner) is performing, a type of tissue being operated on,or a body cavity that is the subject of the surgical procedure. Thesituational awareness system of the surgical hub can be configured toderive the contextual information from the data received from the datasources in a variety of different ways. In an exemplary embodiment, thecontextual information received by the situational awareness system ofthe surgical hub is associated with a particular control adjustment orset of control adjustments for one or more modular devices. The controladjustments each correspond to a variable parameter. In one example, thesituational awareness system includes a pattern recognition system, ormachine learning system (e.g., an artificial neural network), that hasbeen trained on training data to correlate various inputs (e.g., datafrom databases, patient monitoring devices, and/or modular devices) tocorresponding contextual information regarding a surgical procedure. Inother words, a machine learning system can be trained to accuratelyderive contextual information regarding a surgical procedure from theprovided inputs. In another example, the situational awareness systemcan include a lookup table storing pre-characterized contextualinformation regarding a surgical procedure in association with one ormore inputs (or ranges of inputs) corresponding to the contextualinformation. In response to a query with one or more inputs, the lookuptable can return the corresponding contextual information for thesituational awareness system for controlling at least one modulardevice. In another example, the situational awareness system includes afurther machine learning system, lookup table, or other such system,which generates or retrieves one or more control adjustments for one ormore modular devices when provided the contextual information as input.

A surgical hub including a situational awareness system may provide anynumber of benefits for a surgical system. One benefit includes improvingthe interpretation of sensed and collected data, which would in turnimprove the processing accuracy and/or the usage of the data during thecourse of a surgical procedure. Another benefit is that the situationalawareness system for the surgical hub may improve surgical procedureoutcomes by allowing for adjustment of surgical instruments (and othermodular devices) for the particular context of each surgical procedure(such as adjusting to different tissue types) and validating actionsduring a surgical procedure. Yet another benefit is that the situationalawareness system may improve surgeon's and/or other medicalpractitioners' efficiency in performing surgical procedures byautomatically suggesting next steps, providing data, and adjustingdisplays and other modular devices in the surgical theater according tothe specific context of the procedure. Another benefit includesproactively and automatically controlling modular devices according tothe particular step of the surgical procedure that is being performed toreduce the number of times that medical practitioners are required tointeract with or control the surgical system during the course of asurgical procedure, such as by a situationally aware surgical hubproactively activating a generator to which an RF electrosurgicalinstrument is connected if it determines that a subsequent step of theprocedure requires the use of the instrument. Proactively activating theenergy source allows the instrument to be ready for use a soon as thepreceding step of the procedure is completed.

For example, a situationally aware surgical hub can be configured todetermine what type of tissue is being operated on. Therefore, when anunexpectedly high force to close a surgical instrument's end effector isdetected, the situationally aware surgical hub can be configured tocorrectly ramp up or ramp down a motor of the surgical instrument forthe type of tissue, e.g., by changing or causing change of at least onevariable parameter of an algorithm for the surgical instrument regardingmotor speed or torque.

For another example, a type of tissue being operated can affectadjustments that are made to compression rate and load thresholds of asurgical stapler for a particular tissue gap measurement. Asituationally aware surgical hub can be configured to infer whether asurgical procedure being performed is a thoracic or an abdominalprocedure, allowing the surgical hub to determine whether the tissueclamped by an end effector of the surgical stapler is lung tissue (for athoracic procedure) or stomach tissue (for an abdominal procedure). Thesurgical hub can then be configured to cause adjustment of thecompression rate and load thresholds of the surgical staplerappropriately for the type of tissue, e.g., by changing or causingchange of at least one variable parameter of an algorithm for thesurgical stapler regarding compression rate and load threshold.

As yet another example, a type of body cavity being operated in duringan insufflation procedure can affect the function of a smoke evacuator.A situationally aware surgical hub can be configured to determinewhether the surgical site is under pressure (by determining that thesurgical procedure is utilizing insufflation) and determine theprocedure type. As a procedure type is generally performed in a specificbody cavity, the surgical hub can be configured to control a motor rateof the smoke evacuator appropriately for the body cavity being operatedin, e.g., by changing or causing change of at least one variableparameter of an algorithm for the smoke evacuator regarding motor rate.Thus, a situationally aware surgical hub may provide a consistent amountof smoke evacuation for both thoracic and abdominal procedures.

As yet another example, a type of procedure being performed can affectthe optimal energy level for an ultrasonic surgical instrument or radiofrequency (RF) electrosurgical instrument to operate at. Arthroscopicprocedures, for example, require higher energy levels because an endeffector of the ultrasonic surgical instrument or RF electrosurgicalinstrument is immersed in fluid. A situationally aware surgical hub canbe configured to determine whether the surgical procedure is anarthroscopic procedure. The surgical hub can be configured to adjust anRF power level or an ultrasonic amplitude of the generator (e.g., adjustenergy level) to compensate for the fluid filled environment, e.g., bychanging or causing change of at least one variable parameter of analgorithm for the instrument and/or a generator regarding energy level.Relatedly, a type of tissue being operated on can affect the optimalenergy level for an ultrasonic surgical instrument or RF electrosurgicalinstrument to operate at. A situationally aware surgical hub can beconfigured to determine what type of surgical procedure is beingperformed and then customize the energy level for the ultrasonicsurgical instrument or RF electrosurgical instrument, respectively,according to the expected tissue profile for the surgical procedure,e.g., by changing or causing change of at least one variable parameterof an algorithm for the instrument and/or a generator regarding energylevel. Furthermore, a situationally aware surgical hub can be configuredto adjust the energy level for the ultrasonic surgical instrument or RFelectrosurgical instrument throughout the course of a surgicalprocedure, rather than just on a procedure-by-procedure basis. Asituationally aware surgical hub can be configured to determine whatstep of the surgical procedure is being performed or will subsequentlybe performed and then update the control algorithm(s) for the generatorand/or ultrasonic surgical instrument or RF electrosurgical instrumentto set the energy level at a value appropriate for the expected tissuetype according to the surgical procedure step.

As another example, a situationally aware surgical hub can be configuredto determine whether the current or subsequent step of a surgicalprocedure requires a different view or degree of magnification on adisplay according to feature(s) at the surgical site that the surgeonand/or other medical practitioner is expected to need to view. Thesurgical hub can be configured to proactively change the displayed view(supplied by, e.g., an imaging device for a visualization system)accordingly so that the display automatically adjusts throughout thesurgical procedure.

As yet another example, a situationally aware surgical hub can beconfigured to determine which step of a surgical procedure is beingperformed or will subsequently be performed and whether particular dataor comparisons between data will be required for that step of thesurgical procedure. The surgical hub can be configured to automaticallycall up data screens based upon the step of the surgical procedure beingperformed, without waiting for the surgeon or other medical practitionerto ask for the particular information.

As another example, a situationally aware surgical hub can be configuredto determine whether a surgeon and/or other medical practitioner ismaking an error or otherwise deviating from an expected course of actionduring the course of a surgical procedure, e.g., as provided in apre-operative surgical plan. For example, the surgical hub can beconfigured to determine a type of surgical procedure being performed,retrieve a corresponding list of steps or order of equipment usage(e.g., from a memory), and then compare the steps being performed or theequipment being used during the course of the surgical procedure to theexpected steps or equipment for the type of surgical procedure that thesurgical hub determined is being performed. The surgical hub can beconfigured to provide an alert (visual, audible, and/or tactile)indicating that an unexpected action is being performed or an unexpecteddevice is being utilized at the particular step in the surgicalprocedure.

In certain instances, operation of a robotic surgical system, such asany of the various robotic surgical systems described herein, can becontrolled by the surgical hub based on its situational awareness and/orfeedback from the components thereof and/or based on information from acloud (e.g., the cloud 713 of FIG. 18 ).

Embodiments of situational awareness systems and using situationalawareness systems during performance of a surgical procedure aredescribed further in previously mentioned U.S. patent application Ser.No. 16/729,772 entitled “Analyzing Surgical Trends By A Surgical System”filed Dec. 30, 2019, U.S. patent application Ser. No. 16/729,747entitled “Dynamic Surgical Visualization Systems” filed Dec. 30, 2019,U.S. patent application Ser. No. 16/729,744 entitled “VisualizationSystems Using Structured Light” filed Dec. 30, 2019, U.S. patentapplication Ser. No. 16/729,778 entitled “System And Method ForDetermining, Adjusting, And Managing Resection Margin About A SubjectTissue” filed Dec. 30, 2019, U.S. patent application Ser. No. 16/729,729entitled “Surgical Systems For Proposing And Corroborating Organ PortionRemovals” filed Dec. 30, 2019, U.S. patent application Ser. No.16/729,778 entitled “Surgical System For Overlaying Surgical InstrumentData Onto A Virtual Three Dimensional Construct Of An Organ” filed Dec.30, 2019, U.S. patent application Ser. No. 16/729,751 entitled “SurgicalSystems For Generating Three Dimensional Constructs Of Anatomical OrgansAnd Coupling Identified Anatomical Structures Thereto” filed Dec. 30,2019, U.S. patent application Ser. No. 16/729,740 entitled “SurgicalSystems Correlating Visualization Data And Powered Surgical InstrumentData” filed Dec. 30, 2019, U.S. patent application Ser. No. 16/729,737entitled “Adaptive Surgical System Control According To Surgical SmokeCloud Characteristics” filed Dec. 30, 2019, U.S. patent application Ser.No. 16/729,796 entitled “Adaptive Surgical System Control According ToSurgical Smoke Particulate Characteristics” filed Dec. 30, 2019, U.S.patent application Ser. No. 16/729,803 entitled “Adaptive VisualizationBy A Surgical System” filed Dec. 30, 2019, and U.S. patent applicationSer. No. 16/729,807 entitled “Method Of Using Imaging Devices InSurgery” filed Dec. 30, 2019.

Surgical Procedures of the Lung

Various aspects of the devices, systems, and methods described hereinmay relate to a surgical procedure performed on a lung. For example, alung resection, e.g., a lobectomy, is a surgical procedure in which allor part, e.g., one or more lobes, of a lung is removed. The purpose ofperforming a lung resection is to treat a damaged or diseased lung as aresult of, for example, lung cancer, emphysema, or bronchiectasis.

During a lung resection, the lung or lungs are first deflated, andthereafter one or more incisions are made on the patient's side betweenthe patient's ribs to reach the lungs laparoscopically. Surgicalinstruments, such as graspers and a laparoscope, are inserted throughthe incision. Once the infected or damaged area of the lung isidentified, the area is dissected from the lung and removed from the oneor more incisions. The dissected area and the one or more incisions canbe closed, for example, with a surgical stapler or stitches.

Since the lung is deflated during surgery, the lung, or certain portionsthereof, may need to be mobilized to allow the surgical instruments toreach the surgical site. This mobilization can be carried out bygrasping the outer tissue layer of the lung with graspers and applying aforce to the lung through the graspers. However, the pleura andparenchyma of the lung are very fragile and therefore can be easilyripped or torn under the applied force. Additionally, duringmobilization, the graspers can cut off blood supply to one or more areasof the lung.

Further, a breathing tube is placed into the patient's airway to alloweach lung to be separately inflated during surgery. Inflation of thelung can cause the lung to move and match pre-operative imaging and/orallow the surgeon to check for leaks at the dissected area(s). However,by inflating the whole lung, working space is lost around the lung dueto the filling of the thoracic cavity. Additionally, inflating a wholelung can take time and does not guarantee easy leak detection ifmultiple portions of the lung are operated on during the surgicalprocedure.

Surgical Procedures of the Colon

Various aspects of the devices, systems, and methods described hereinmay relate to a surgical procedure performed on a colon. For example,surgery is often the main treatment for early-stage colon cancers. Thetype of surgery used depends on the stage (extent) of the cancer, whereit is in the colon, and the goal of the surgery. Some early coloncancers (stage 0 and some early stage I tumors) and most polyps can beremoved during a colonoscopy. However, if the cancer has progressed, alocal excision or colectomy may be required. A colectomy is surgery toremove all or part of the colon. In certain instances, nearby lymphnodes are also removed. If only part of the colon is removed, it'scalled a hemicolectomy, partial colectomy, or segmental resection inwhich the surgeon takes out the diseased part of the colon with a smallsegment of non-diseased colon on either side. Usually, about one-fourthto one-third of the colon is removed, depending on the size and locationof the cancer. Major resections of the colon are illustrated in FIG.21A, in which A-B is a right hemicolectomy, A-C is an extended righthemicolectomy, B-C is a transverse colectomy, C-E is a lefthemicolectomy, D-E is a sigmoid colectomy, D-F is an anterior resection,D-G is a (ultra) low anterior resection, D-H is an abdomino-perinealresection, A-D is a subtotal colectomy, A-E is a total colectomy, andA-H is a total procto-colectomy. Once the resection is complete, theremaining intact sections of colon are then reattached.

A colectomy can be performed through an open colectomy, where a singleincision through the abdominal wall is used to access the colon forseparation and removal of the affected colon tissue, and through alaparoscopic-assisted colectomy. With a laparoscopic-assisted colectomy,the surgery is done through many smaller incisions with surgicalinstruments and a laparoscope passing through the small incisions toremove the entire colon or a part thereof. At the beginning of theprocedure, the abdomen is inflated with gas, e.g., carbon dioxide, toprovide a working space for the surgeon. The laparoscope transmitsimages inside the abdominal cavity, giving the surgeon a magnified viewof the patient's internal organs on a monitor or other display. Severalother cannulas are inserted to allow the surgeon to work inside andremove part(s) of the colon. Once the diseased parts of the colon areremoved, the remaining ends of the colon are attached to each other,e.g., via staplers or stitches. The entire procedure may be completedthrough the cannulas or by lengthening one of the small cannulaincisions.

During a laparoscopic-assisted colectomy procedure, it is oftendifficult to obtain an adequate operative field. Oftentimes, dissectionsare made deep in the pelvis which makes it difficult to obtain adequatevisualization of the area. As a result, the lower rectum must be liftedand rotated to gain access to the veins and arteries around both sidesof the rectum during mobilization. During manipulation of the lowerrectum, bunching of tissue and/or overstretching of tissue can occur.Additionally, a tumor within the rectum can cause adhesions in thesurrounding pelvis, and as a result, this can require freeing the rectalstump and mobilizing the mesentery and blood supply before transectionand removal of the tumor.

Further, multiple graspers are needed to position the tumor for removalfrom the colon. During dissection of the colon, the tumor should beplaced under tension, which requires grasping and stretching thesurrounding healthy tissue of the colon. However, the manipulating ofthe tissue surrounding the tumor can suffer from reduced blood flow andtrauma due to the graspers placing a high grip force on the tissue.Additionally, during a colectomy, the transverse colon and upperdescending colon may need to be mobilized to allow the healthy, goodremaining colon to be brought down to connect to the rectal stump afterthe section of the colon containing the tumor is transected and removed.

After a colectomy, the remaining healthy portions of the colon must bereattached to one another to create a path for waste to leave the body.However, when using laparoscopic instruments to perform the colectomy,one single entry port may not have a large enough range of motion tomove the one end of the colon to a connecting portion of the colon. Assuch, a second entry port is therefore needed to laparoscopically insertsurgical instruments to help mobilize the colon in order to properlyposition the colon.

Surgical Procedures of the Stomach

Various aspects of the devices, systems, and methods described hereinmay relate to a surgical procedure performed on a stomach. For example,surgery is the most common treatment for stomach cancer. When surgery isrequired for stomach cancer, the goal is to remove the entire tumor aswell as a good margin of healthy stomach tissue around the tumor.Different procedures can be used to remove stomach cancer. The type ofprocedure used depends on what part of the stomach the cancer is locatedand how far it has grown into nearby areas. For example, endoscopicmucosal resection (EMR) and endoscopic submucosal dissection (ESD) areprocedures on the stomach can be used to treat some early-stage cancers.These procedures do not require a cut in the skin, but instead thesurgeon passes an endoscope down the throat and into the stomach of thepatient. Surgical tools (e.g., MEGADYNE™ Tissue Dissector orElectrosurgical Pencils) are then passed through the working channel ofthe endoscope to remove the tumor and some layers of the normal stomachwall below and around it.

Other surgical procedures performed on a stomach include a subtotal(partial) or a total gastrectomy that can be performed as an openprocedure. e.g., surgical instruments are inserted through a largeincision in the skin of the abdomen, or as a laparoscopic procedure,e.g., surgical instruments are inserted into the abdomen through severalsmall cuts. For example, a laparoscopic gastrectomy procedure generallyinvolves insufflation of the abdominal cavity with carbon dioxide gas toa pressure of around 15 millimeters of mercury (mm Hg). The abdominalwall is pierced and a straight tubular cannula or trocar, such as acannula or trocar having a diameter in a range of about 5 mm to about 10mm, is then inserted into the abdominal cavity. A laparoscope connectedto an operating room monitor is used to visualize the operative fieldand is placed through one of the trocar(s). Laparoscopic surgicalinstruments are placed through two or more additional cannulas ortrocars for manipulation by medical practitioner(s), e.g., surgeon andsurgical assistant(s), to remove the desired portion(s) of the stomach.

In certain instances, laparoscopic and endoscopic cooperative surgerycan be used to remove gastric tumors. This cooperative surgery typicallyinvolves introduction of an endoscope, e.g., a gastroscope, andlaparoscopic trocars. A laparoscope and tissue manipulation anddissection surgical instruments are introduced through the trocar. Thetumor location can be identified via the endoscope and a cutting elementthat is inserted into the working channel of the endoscope is then usedfor submucosal resection around the tumor. A laparoscopic dissectionsurgical instrument is then used for seromuscular dissection adjacentthe tumor margins to create an incision through the stomach wall. Thetumor is then pivoted through this incision from the intraluminal space,e.g., inside the stomach, to the extraluminal space, e.g., outside ofthe stomach. A laparoscopic surgical instrument, e.g., an endocutter,can be used to then complete the transection of the tumor from thestomach wall and seal the incision.

Surgical Procedures of the Intestine

Various aspects of the devices, systems, and methods described hereinmay relate to a surgical procedure performed on an intestine. Forexample, a duodenal mucosal resurfacing (DMR) procedure can be performedendoscopically to treat insulin-resistant metabolic diseases such astype 2 diabetes. The DMR procedure can be an effective treatment becauseit affects detection of food. The DMR procedure inhibits duodenumfunction such that food tends to be sensed deeper in the intestine thannormal, e.g., sensed after passage through the duodenum (which is thefirst part of the small intestine). The patient's body thus senses sugardeeper in the intestine than is typical and thus reacts to the sugarlater than is typical such that glycemic control can be improved. Theirregular function of the duodenum changes the body's typical responseto the food and, through nervous system and chemical signals, causes thebody to adapt its response to the glucose level to increase insulinlevels.

In the DMR procedure, the duodenal mucosa is lifted, such as withsaline, and then the mucosa is ablated, e.g., using an ablation deviceadvanced into the duodenum through a working channel of an endoscope.Lifting the mucosa before ablation helps protect the duodenum's outerlayers from being damaged by the ablation. After the mucosa is ablated,the mucosa later regenerates. Examples of ablation devices are NeuWave™ablation probes (available from Ethicon US LLC of Cincinnati, Ohio).Another example of an ablation device is the Hyblate catheter ablationprobe (available from Hyblate Medical of Misgav, Israel). Anotherexample of an ablation device is the Barxx™ HaloFlex (available fromMedtronic of Minneapolis, Minn.).

FIG. 21B illustrates one embodiment of a DMR procedure. As shown in FIG.21B, a laparoscope 1400 is positioned external to a duodenum 1402 forexternal visualization of the duodenum 1402. An endoscope 1404 isadvanced transorally through an esophagus 1406, through a stomach 1408,and into the duodenum 1402 for internal visualization of the duodenum1402. An ablation device 1410 is advanced through a working channel ofthe endoscope 1404 to extend distally from the endoscope 1404 into theduodenum 1402. A balloon 1412 of the ablation device 1410 is shownexpanded or inflated in FIG. 21B. The expanded or inflated balloon 1412can help center the ablation device's electrode so even circumferentialablating can occur before the ablation device 1410 is advanced and/orretracted to repeat ablation. Before the mucosa is ablated using theablation device 1410, the duodenal mucosa is lifted, such as withsaline. In some embodiments in addition to or instead of including theballoon 1412, the ablation device 1410 can be expandable/collapsibleusing an electrode array or basket configured to expand and collapse.

The laparoscope's external visualization of the duodenum 1402 can allowfor thermal monitoring of the duodenum 1402, which may help ensure thatthe outer layers of the duodenum 1402 are not damaged by the ablation ofthe duodenal mucosa, such as by the duodenum being perforated. Variousembodiments of thermal monitoring are discussed further, for example,below and in U.S. Pat. App No. 63/249,658 entitled “Surgical Devices,Systems, And Methods For Control Of One Visualization With Another”filed on Sep. 29, 2021. The endoscope 1404 and/or the ablation device1410 can include a fiducial marker thereon that the laparoscope 1400 canbe configured to visualize through the duodenum's tissue, e.g., by usinginvisible light, to help determine where the laparoscope 1400 shouldexternally visualize the duodenum 1402 at a location where ablationoccurs. Various embodiments of fiducial markers are discussed further,for example, in U.S. Pat. App No. 63/249,652 entitled “Surgical Devices,Systems, Methods Using Fiducial Identification And Tracking” filed onSep. 29, 2021 and in U.S. Pat. App No. 63/249,658 entitled “SurgicalDevices, Systems, And Methods For Control Of One Visualization WithAnother” filed on Sep. 29, 2021.

Control of Cooperative Surgical Instruments

In various aspects, the present disclosure provides methods, devices,and systems for the control of cooperative surgical instruments.

For example, in one embodiment, a system can include first and secondsurgical instruments and a controller. The first and second surgicalinstruments can each be configured to operate on tissue at a firstsurgical treatment site located within a patient, and the controller canbe configured to receive data related to the first instrument from thesecond instrument. Based on the received data, the controller candetermine at least one measured parameter of the first surgicalinstrument and can determine an adjustment of at least one operationalparameter of the first or second surgical instrument based on themeasured parameter.

For example, FIG. 22 provides a schematic of one exemplary surgicalsystem 1000 that can provide for cooperative control of surgicalinstruments regarding locations and movements of various instruments. Asshown, the system 1000 includes a first surgical instrument 1010configured to be inserted into a patient, a first endoscope 1020configured to be inserted into the patient and to visualize the firstsurgical instrument 1010, a second surgical instrument 1030 configuredto be inserted into the patient, and a second endoscope 1040 configuredto be inserted into the patient and to visualize the second surgicalinstrument 1030. The first surgical instrument 1010, the first endoscope1020, the second surgical instrument 1040, and the second endoscope 1040are each configured to be in operable communication with a controller1050 of the system 1000 that is configured to control the operation ofthe first and second surgical instruments 1010, 1030 based on datareceived from at least one of the first and second surgical instruments1010, 1030 and the first and second endoscopes 1020, 1040. The first andsecond surgical instruments 1010, 1030 are configured to be insertedinto a shared body cavity or separate body cavities separated by acommon tissue wall. While the system 1000 includes the first and secondendoscopes 1020, 1040, in some embodiments, one or both of theendoscopes 1020, 1040 can be omitted from the system 1000.

The first surgical instrument 1010 and the second surgical instrument1030 can each be any suitable surgical device configured to manipulateand/or treat tissue. The first surgical instrument 1010 and the secondsurgical instrument 1030 can each be similar to the surgical device 102of FIG. 1 , the surgical device 202 of FIG. 8 , or other surgical devicedescribed herein. As mentioned above, examples of surgical devicesinclude a surgical dissector, a surgical stapler, a surgical grasper, aclip applier, a smoke evacuator, a surgical energy device (e.g.,mono-polar probes, bi-polar probes, ablation probes, an ultrasounddevice, an ultrasonic end effector, etc.), various other smart poweredhand-held devices, etc. For example, in some embodiments, the firstsurgical instrument 1010 and/or the second surgical instrument 1030 caninclude an end effector having opposing jaws that extend from a distalend of a shaft of the surgical device and that are configured to engagetissue therebetween.

The first endoscope 1020 and the second endoscope 1040 can each includean imaging device configured to acquire an image of a surgical site in aminimally invasive surgical procedure. The first endoscope 1020 and thesecond endoscope 1040 can each be similar to the imaging device 120 ofFIG. 1 , the imaging device 220 of FIG. 8 , or other imaging devicedescribed herein. Although some implementations of the current subjectmatter are described herein as using one or more endoscopes to acquireimages of the surgical site, any type of scope suitable for use in aminimally invasive surgical procedure can be used in conjunction withthe systems, methods, and devices described herein. As mentioned above,examples of scopes include an arthroscope, an angioscope, abronchoscope, a choledochoscope, a colonoscope, a cytoscope, aduodenoscope, an enteroscope, an esophagogastro-duodenoscope(gastroscope), a laryngoscope, a nasopharyngo-neproscope, asigmoidoscope, a thoracoscope, an ureteroscope, and an exoscope. One ormore of these exemplary types of scopes can be used together in aminimally invasive surgical procedure in any feasible combination.

The controller 1050 includes a processor 1051 configured to perform oneor more of the operations described herein and a memory 1052 that isconfigured to store instructions for causing the processor 1051 toperform the operations. The controller 1050 also includes a firstsurgical instrument interface 1053, a first endoscope interface 1054, asecond surgical instrument interface 1055, and a second endoscopeinterface 1056. As shown in FIG. 22 , the first surgical instrument 1010is coupled to the controller 1050 via the first surgical instrumentinterface 1053 and as such can receive movement and actuationinstructions from the processor 1051. The first endoscope 1020 iscoupled to the controller 1050 via the first endoscope interface 1054,and as such can provide data characterizing images acquired by the firstendoscope 1020 to the processor 1051, and/or the memory 1052 for lateruse by the processor 1051 in performing the operations described herein.Similar to the first surgical instrument 1010, the second surgicalinstrument 1030 is coupled to the controller 1050 via the secondsurgical instrument interface 1055 and as such can receive movement andactuation instructions from the processor 1051. Similar to the firstendoscope 1020, the second endoscope 1040 is coupled to the controller1050 via the second endoscope interface 1056 and as such can providedata characterizing images acquired by the second endoscope 1040 to theprocessor 1051 and/or the memory 1052 for later use by the processor1051 in performing the operations described herein. In some embodiments,each of the first surgical instrument interface 1053, the firstendoscope interface 1054, the second surgical instrument interface 1055,and the second endoscope interface 1056 may be different from oneanother so as to accommodate differences between the controllerinterfaces of various ones of the first surgical instrument 1010, thefirst endoscope 1020, the second surgical instrument 1030, and thesecond endoscope 1040. Thus, the first and second surgical instruments1010, 1030 and the first and second endoscopes 1020, 1040 can all beindependently controlled through their various interfaces or can bejointly controlled in a number of different ways, as discussed furtherbelow.

As shown, the system 1000 can also include a display 1060 that isoperably coupled to the controller 1050 and configured to graphicallydepict the images acquired by one or more of the first endoscope 1020and the second endoscope 1040. In some embodiments, the controller 1050can receive a stream of image data from each of the first endoscope 1020and the second endoscope 1040, determine an image and/or video feed inreal-time from the received image data, and provide the image and/orvideo feed to the display 1060 for depiction thereon and viewing by auser. The system 1000, the controller 1050, and/or the display 1060 canbe incorporated into various robotic surgical systems and/or can be partof a surgical hub or other computer system, as discussed above.

Furthermore, in some embodiments, the system 1000 can allow for singleuser control of multiple surgical devices, such as the first and secondsurgical instruments 1010, 1030 and the first and second endoscopes1020, 1040. For example, control of the system 1000 or individualinstruments 1010, 1030 and individual endoscopes 1020, 1040 can betethered to various surgeon control tablets, laparoscopic robotconsoles, other instruments such as smart powered hand held instruments,etc., illustrated as a master control interface 2000 in FIG. 22 , toprovide cooperative control of multiple independently-controllabledevices from a single device. In an exemplary embodiment, the mastercontrol interface 2000 is a surgical hub, although the master controlinterface 2000 can be configured as another type of computer system.

Because the controller 1050 of the system 1000 communicates with themaster control interface 2000, the master control interface 2000 isconfigured to receive user inputs for controlling each of the firstsurgical instrument 1010, the first endoscope 1020, the second surgicalinstrument 1030, and the second endoscope 1040 and to communicateinstructions to the controller 1050 based on the received user inputs.Thus, a user can control all of the first surgical instrument 1010, thefirst endoscope 1020, the second surgical instrument 1030, and thesecond endoscope 1040 from a single interface, the master controlinterface 2000, which may achieve significant flexibility duringperformance of surgical procedures. Additionally, separate control overthe system 1000 using the master control interface 2000, such as by asurgeon within the sterile field, may allow for safer procedures byproviding for a number of different control configurations based on eachprocedure's and/or each user's needs.

For example, in some embodiments, the user can toggle the master controlinterface 2000 to access one or more of the first surgical instrumentinterface 1053, the first endoscope interface 1054, the second surgicalinstrument interface 1055, and the second endoscope interface 1056 andtoggle between the different interfaces 1053, 1054, 1055, 1056. Thevarious interfaces 1053, 1054, 1055, 1056 can retain their normalappearance, or can have a customized appearance for interaction via themaster control interface 2000. The master control interface 2000 canprovide access to one or more of the interfaces 1053, 1054, 1055, 1056.In other embodiments, a first user (such as a surgeon) can control theinstruments 1010, 1030 and the endoscopes 1020, 1040 during performanceof a surgical procedure through the master control interface 2000, whilea second user (such as a second surgeon or assistant) can controlmovement of one of the instruments 1010, 1030 or endoscopes 1020, 1040during certain stages of the surgical procedure, such as insertion orremoval, through one or more of the interfaces 1053, 1054, 1055, 1056.Thus, the first user can take over control of one or more of theinstruments 1010, 1030 or endoscopes 1020, 1040 at more challenging ordifficult points during a procedure while allowing the second user tocontrol one or more of the instruments 1010, 1030 or endoscopes 1020,1040 during simpler points of the procedure.

For example, during use of circular staplers or endocutters, such asduring colorectal surgery, a surgeon may need to operate within asterile field while having an assistant control positioning, clamping,and/or firing of instruments from outside the sterile field to create afinal anastomosis. During such a procedure, the surgeon and theassistant thus must coordinate with each other from inside and outsidethe sterile field, which can cause inefficiencies and complications. Assuch, in one exemplary embodiment, the instruments 1010, 1030 used inthe system 1000 can be circular staplers 1010 a, 1030 a illustrated inFIG. 23 . Each stapler 1010 a, 1030 a has an anvil 1012 a, 1032 a and areplaceable staple cartridge 1014 a, 1034 a disposed at a distal end ofa shaft 1016 a, 1036 a, and each stapler 1010 a, 1030 a also has acircular anvil retraction/release motor and a firing control motor. Asdiscussed above with respect to the system 1000 and the first and secondinstruments 1010, 1030, the staplers 1010 a, 1030 a are each operablycoupled to the master control interface 2000 to allow separate accessand control of one or more of the functionalities of the staplers 1010a, 1030 a by the master control interface 2000 either from within thesterile field with the patient or from outside of the sterile field.When the surgeon has the master control interface 2000 within thesterile field, the assistant outside the field simply needs to positionthe stapler(s) 1010 a, 1030 a and maintain a particular position whilethe surgeon inside the sterile field is able to clamp down on tissue toa desired level and fire the stapler while still being able to observeoperational elements in real time, such as micro tissue tension anddirect operation of the corresponding stapler 1010 a, 1030 a to ensurethe instrument is working correctly.

In still other embodiments, the instruments 1010, 1030 workcooperatively together to provide tracking and placement assistance ofone or both of the instruments 1010, 1030. For example, in someembodiments, the first instrument 1010 is an endocutter jaw without atether or shaft during positioning, which makes correctly positioningthe first instrument 1010 difficult. One or both of the endoscopes 1020,1040 track a position of the first instrument 1010 and augments an imageof the position, such as on the display 1060, over anatomy that mayobscure the first instrument 1010 to provide information on properpositioning and allow a separate shaft or other drive function mechanismto be correctly positioned relative to the endocutter jaw of the firstinstrument 1010.

Furthermore, in still another embodiment, when the system 1000 is partof a robotic surgical system, the first user selects a visual feed fromone of the endoscopes 1020, 1040, such as an endoluminal endoscope or abronchoscope, that is controlled by the second user during insertion ofthe first or second endoscope 1020, 1040 and prior to docking the firstor second endoscope 1020, 1040. A view of where the selected endoscope1020, 1040 is within the patient is augmented onto a display, such asthe display 1060, so that the first user has the ability to direct whenthe selected endoscope 1020, 1040 is docked and controllable through therobotic surgical system, such as when in proximity to a clinical job ortumor position.

Additional details regarding various embodiments of augmented and mergedimages and of tracking surgical instruments are provided in, forexample, previously mentioned U.S. App. No. 63/249,980 entitled“Cooperative Access” filed on Sep. 29, 2021. Additional embodiments ofvarious surgical staplers and their corresponding functionalities arefurther described in U.S. Pat. No. 10,492,788, entitled “PoweredSurgical Circular Stapler With Removable Cartridge And Variable StapleHeight” issued Dec. 3, 2019, U.S. Pat. Pub. No. 2018/0353174 filed Jun.13, 2017 and entitled “Surgical Stapler with Controlled Healing,” U.S.Pat. No. 10,569,071 entitled “Medicant Eluting Adjuncts And Methods OfUsing Medicant Eluting Adjuncts” issued Feb. 25, 2020, U.S. Pat. No.10,716,564 entitled “Stapling Adjunct Attachment” issued Jul. 21, 2020,U.S. Pat. Pub. No. 2013/0256377 entitled “Layer Comprising DeployableAttachment Members” filed Feb. 8, 2013, U.S. Pat. No. 8,393,514 entitled“Selectively Orientable Implantable Fastener Cartridge” filed Sep. 30,2010, U.S. Pat. No. 8,317,070 entitled “Surgical Stapling Devices ThatProduce Formed Staples Having Different Lengths” filed Feb. 28, 2007,U.S. Pat. No. 7,143,925 entitled “Surgical Instrument Incorporating EAPBlocking Lockout Mechanism” filed Jun. 21, 2005, U.S. Pat. Pub. No.2015/0134077 entitled “Sealing Materials For Use In Surgical Stapling”filed Nov. 8, 2013, U.S. Pat. Pub. No. 2015/0134076, entitled “HybridAdjunct Materials for Use in Surgical Stapling” filed on Nov. 8, 2013,U.S. Pat. Pub. No. 2015/0133996 entitled “Positively Charged ImplantableMaterials and Method of Forming the Same” filed on Nov. 8, 2013, U.S.Pat. Pub. No. 2015/0129634 entitled “Tissue Ingrowth Materials andMethod of Using the Same” filed on Nov. 8, 2013, U.S. Pat. Pub. No.2015/0133995 entitled “Hybrid Adjunct Materials for Use in SurgicalStapling” filed on Nov. 8, 2013, U.S. Pat. Pub. No. 2015/0272575entitled “Surgical Instrument Comprising a Sensor System” and filed onMar. 26, 2014, U.S. Pat. Pub. No. 2015/0351758 entitled “AdjunctMaterials and Methods of Using Same in Surgical Methods for TissueSealing” filed on Jun. 10, 2014, U.S. Pat. Pub. No. 2013/0146643entitled “Adhesive Film Laminate” filed Feb. 8, 2013, U.S. Pat. No.7,601,118 entitled “Minimally Invasive Medical Implant And InsertionDevice And Method For Using The Same” filed Sep. 12, 2007, and U.S. Pat.Pub. No. 2013/0221065 entitled “Fastener Cartridge Comprising AReleasably Attached Tissue Thickness Compensator” filed Feb. 8, 2013,which are hereby incorporated by reference in their entireties.

Another exemplary embodiment in which separate access and control overinstruments by the master control interface 2000 may be beneficialinvolves the application of mono-polar or bi-polar energy to tissueduring performance of a surgical procedure. In an exemplary procedure,mono-polar energy moves from a high energy density tip of an instrumentwhere electro-cautery occurs to various diffuse return pads affixed to apatient's skin. However, a path between where the instrument tip is andthe return pads can become isolated and restricted for various reasons,such as if the path extends along a narrowing artery or vein. In suchprocedures, unintended secondary cautery can occur between theinstrument tip and the area of restriction. As such, in anotherexemplary embodiment, the instruments 1010, 1030 used in the system 1000can be electrosurgical instruments 1010 b, 1030 b illustrated in FIG. 24. The electrosurgical instruments 1010 b, 1030 b can have a variety ofconfigurations. In this illustrated embodiment, each electrosurgicalinstrument 1010 b, 1030 b has a first jaw 1012 b, 1032 b and a secondjaw 1014 b, 1034 b disposed at a distal end of a shaft 1016 b, 1036 b,and each jaw 1012 b, 1014 b, 1032 b, 1034 b has an electrode thereon.Each electrosurgical instrument 1010 b, 1030 b can also selectivelyapply mono-polar or bi-polar energy to target treatment tissue. Asdiscussed above with respect to the system 1000 and the first and secondinstruments 1010, 1030, the electrosurgical instruments 1010 b, 1030 bare each operatively coupled to the master control interface 2000, viathe controller 1050, to allow control of one or more of thefunctionalities of the electrosurgical instruments 1010 b, 1030 b by themaster control interface 2000, such as actuating energy delivery andmonitoring active and return paths of energy from each instrument 1010b, 1030 b. When a user intends to apply mono-polar energy but apotential area of restriction is detected, before applying mono-polarenergy from one of the electrosurgical instruments 1010 b, 1030 b, theuser can select the other electrosurgical instrument 1010 b, 1030 b as asubstitute return path instead of the return pads affixed to thepatient's skin to avoid unintended cautery. Additional embodiments ofvarious electrosurgical instruments, their correspondingfunctionalities, and sensors incorporated therein are further describedin U.S. Pat. Pub. No. 2009/0062792, entitled “Electrical AblationSurgical Instruments” and published Mar. 5, 2009, U.S. Pat. Pub. No.2019/0125431, entitled “Surgical suturing instrument configured tomanipulate tissue using mechanical and electrical power” and publishedMay 2, 2019, U.S. Pat. Pub. No. 2017/0202591, entitled “Modular batterypowered handheld surgical instrument with selective application ofenergy based on tissue characterization” and published Jul. 20, 2017,U.S. Pat. No. 9,757,128, entitled “Multiple sensors with one sensoraffecting a second sensor's output or interpretation” and issued Sep.12, 2017, U.S. Pat. No. 8,753,338, entitled “Electrosurgical InstrumentEmploying A Thermal Management System” and issued on Jun. 17, 2014, andU.S. Pat. No. 5,558,671, entitled “Impedance feedback monitor forelectrosurgical instrument” and issued on Sep. 24, 1996, which arehereby incorporated by reference in their entireties.

With reference again to FIG. 22 , in still other embodiments, outputfrom the instruments 1010, 1030 and the endoscopes 1020, 1040 can bedisplayed on and controlled through their own interfaces 1053, 1054,1055, 1056 while the master control interface 2000 receives outputand/or images from the instruments 1010, 1030 and the endoscopes 1020,1040, via the controller 1050, without allowing any control of theinstruments 1010, 1030 and the endoscopes 1020, 1040 from the mastercontrol interface 2000. However, in such embodiments, a user can use themaster control interface 2000 to request control of the desired one(s)of the instruments 1010, 1030 and the endoscopes 1020, 1040 and makeadjustments to the instruments 1010, 1030 and the endoscopes 1020, 1040for which control was requested while leaving the overall system 1000under the control of another user. The system 1000 can thus maintaincontrol over all of the instruments 1010, 1030 and the endoscopes 1020,1040 while still accepting adjustments by the master control interface2000.

For example, the master control interface 2000 transmits a query to thesystem 1000 regarding a user input to the master control interface 2000requesting a motion adjustment or actuation of one or more of theinstruments 1010, 1030 and endoscopes 1020, 1040 for which control wasrequested and granted. In response to receiving the query, the system1000 determines whether the requested motion adjustment or actuation isacceptable. Whether or not the requested motion adjustment or actuationis acceptable depends on the current procedure and stage within eachprocedure. The requested motion adjustment or actuation is consideredacceptable in the illustrated embodiment if the request does notcontradict an earlier request made to the system 1000 via one or more ofthe interfaces 1053, 1054, 1055, 1056. Additionally or alternatively,the requested motion adjustment or actuation can be deemed acceptable ifthe request does not violate one or more predetermined rules or limitsset in place on the system 1000, such as specific surgical regions inwhich movement is prevented, certain periods during a procedure whenactuation of functionality on one or more of the instruments 1010, 1030is prevented, etc. As one example, during use of the staplers 1010 a,1030 a discussed above, if the master control interface 2000 requestsfiring of either stapler 1010 a, 1030 a, the system 1000, e.g., thecontroller 1050 thereof, determines if the corresponding staplecartridge 1014 a, 1034 a has been fired yet. If the corresponding staplecartridge 1014 a, 1034 a is determined to not have been fired yet, therequested firing is deemed acceptable, and if the corresponding staplecartridge 1014 a, 1034 a is determined to have been fired and has notyet been replaced, the requested firing is deemed unacceptable.

If determined to be acceptable, the system 1000, e.g., the controller1050 thereof, allows the requested motion adjustment or actuation tooccur. If determined to not be acceptable, the system 1000, e.g., thecontroller 1050 thereof, prevents the requested motion adjustment oractuation from happening, and the master control interface 2000 providesa notification to the user at the master control interface 2000 that therequested motion adjustment or actuation was prevented. The user maytherefore be able to take any desired corrective action that will thenallow the requested motion adjustment or actuation to be allowed and/orto compensate for the requested motion adjustment or actuation nothaving occurred.

In some embodiments, one or more of the surgical instruments 1010, 1030and/or the endoscopes 1020, 1040 can monitor or sense one or moreaspects, further discussed below, of one or more of the other surgicalinstruments 1010, 1030 and/or the endoscopes 1020, 1040, and the system1000, e.g., the controller 1050 thereof, can use the monitored or sensedaspect(s) in a variety of different ways. For example, the monitored orsensed aspect(s) can be communicated throughout the system 1000 and/orto the user such as by information regarding the monitored or sensedaspect(s) being provided on the display 1060, the monitored or sensedaspect(s) can be used to confirm operation of the one or more monitoredor sensed surgical instruments 1010, 1030 and/or endoscopes 1020, 1040,and/or the monitored or sensed aspect(s) can be used to alter operationof the one or more surgical instruments 1010, 1030 and/or endoscopes1020, 1040 that are monitoring the others. For example, a first one ofthe instruments 1010, 1030 and endoscopes 1020, 1040 can monitor asecond one of the instruments 1010, 1030 and endoscopes 1020, 1040regarding one or more aspects of the second one of the instruments 1010,1030 and endoscopes 1020, 1040, and the controller 1050 can then alterone or more functionalities of the second one of the instruments 1010,1030 or endoscopes 1020, 1040 based on the monitored information, asdiscussed below. As such, in some surgical procedures, variousthresholds regarding force, energy applied, etc. can be selected foreach instrument and/or as a total cumulative amount to avoid anyinadvertent tissue trauma. The various pieces of data monitored andcommunicated by and between each of the surgical instruments 1010, 1030and endoscopes 1020, 1040, further discussed below, can be communicatedin a variety of different ways, such as directly between twoinstruments/endoscopes, through the system 1000, using otherinstruments/endoscopes as relays, various additional communication linksor connected hubs, etc.

For example, in some embodiments, one of the endoscopes 1020, 1040 canvisually monitor one of the surgical instruments 1010, 1030 to confirm astate or correct operation of the one of the surgical instruments 1010,1030, and the controller 1050 can alter functionality of the system 1000based on the visual observations. In an exemplary embodiment, theinstruments 1010, 1030 used in the system 1000 are the circular staplers1010 a, 1030 a illustrated in FIG. 23 . As mentioned above, each stapler1010 a, 1030 a has a replaceable stapler cartridge 1014 a, 1034 a.During a first phase of a surgical procedure, one or both of theendoscopes 1020, 1040 visually monitors one or both of the circularstaplers 1010 a, 1030 a to allow confirmation that the cartridge 1014 a,1034 a is present in the corresponding stapler 1010 a, 1030 a. Thecontroller 1050 receives image data from the monitoring one(s) of theendoscope(s) 1020, 1040 and analyzes the received image data todetermine if the visualized stapler(s) 1010 a, 1030 a visualized in theimage data has the cartridge 1014 a, 1034 a present, such as bydetecting whether a certain color is present within the cartridge jaw ascartridges compatible with the stapler have a known color, by detectingwhether a visual pattern of staple cavities is present so as to indicatepresence of the cartridge 1014 a, 1034 a, etc. Based on thedetermination, the controller 1050 can take a variety of differentactions, such as alerting a user (e.g., visual notification via thedisplay 1060, audible notification, and/or tactile notification) if thecartridge 1014 a, 1034 a is determined to not be present in thecorresponding stapler 1010 a, 1030 a, automatically disabling any firingfunctionality of the relevant stapler 1010 a, 1030 a if the cartridge1014 a, 1034 a is determined to not be present in the stapler 1010 a,1030 a, etc. During a second phase of the surgical procedure, themonitoring endoscope(s) 1020, 1040 continue to visually track the one orboth of the staplers 1010 a, 1030 a, and the system 1000, e.g., thecontroller 1050 thereof, detects and tracks when the tracked stapler(s)1010 a, 1030 a has been fired and the corresponding cartridge 1014 a,1034 a has been spent based on the images gathered by the monitoringendoscope(s) 1020, 1040. The system 1000, e.g., the controller 1050thereof, can take a variety of different actions upon determining thatthe cartridge 1014 a, 1034 a has been spent, such as alerting a user(e.g., visual notification via the display 1060, audible notification,and/or tactile notification), disabling any firing functionality of therelevant stapler 1010 a, 1030 a, until the system 1000, e.g., thecontroller 1050 thereof, detects that the spent cartridge 1014 a, 1034 ahas been replaced with a new cartridge, until the system 1000, e.g., thecontroller 1050 thereof, detects that the stapler 1010 a, 1030 a inquestion has been removed from the patient so that a new cartridge canbe loaded, etc. In other embodiments, the endoscope(s) 1020, 1040 canmonitor the stapler(s) 1010 a, 1030 a to determine if the correspondinganvil 1012 a, 1032 a is open or closed, to determine a tissue contactlocation within the anvil 1012 a, 1032 a, etc., and in still otherembodiments in which a surgical stapler is used with linear first andsecond jaws, such as those disclosed in previously mentioned U.S. Pat.No. 10,569,071, the controller 1050 can determine if the jaws are openor closed, a tissue contact location within the jaws, etc.

In another embodiment, during colorectal surgery as discussed above, thesystem 1000 can incorporate one or more additional imaging devices toobserve any distention of a patient's bowel from a laparoscopic side ofa surgical treatment site, indicating to the controller 1050 that theanvil(s) 1012 a, 1032 a on the corresponding stapler(s) 1010 a, 1030 ais open such that the stapler 101 a, 1030 s is ready to be fired, thusconfirming readiness for removal through anastomosis. In still anotherembodiment, rotation or extension of one of the anvils 1012 a, 1032 a ispossible with respect to the housing of the corresponding circularstapler 1010 a, 1030 a so that, as a user rotates the stapler 1010 a,1030 a, engagement of the anvil 1012 a, 1032 a is tightened or loosenedby the controller 1050 to allow the anvil 1012 a, 1032 a to freelyrotate (loosened) or to prevent rotation (tightened) relative to thehousing of the stapler 1010 a, 1030 a to maintain a correct alignment ofthe anvil 1012 a, 1032 a with the cartridge.

In other embodiments, the surgical instruments 1010, 1030 can monitoreach other directly through one or more sensors, as discussed below, andthe system 1000, e.g., the controller 1050 thereof, can alter operationof one or both of the surgical instruments 1010, 1030 based on variousmonitored factors. For example, the instruments 1010, 1030 used in thesystem 1000 can be the electrosurgical instruments 1010 b, 1030 bdiscussed above and illustrated in FIG. 24 . Each instrument 1010 b,1030 b can be configured to monitor the other instrument 1010 b, 1030 bduring use, using one or more sensors, and can therefore, as discussedfurther below, assist in detecting problems with the other instrument1010 b, 1030 b, cooperate to deliver combined electrical loads and/orreturn paths, adjust various settings or functionality of the otherinstrument such as RF power or voltage, detect status of an instrumentsuch as an energized state, and/or sense tissue or parameters of tissuebetween the instruments 1010 b, 1030 b, such as impedance as discussedbelow to more effectively deliver energy.

For example, the second instrument 1030 b can be configured to assist inselectively interrupting or redirecting power transmitted by the firstinstrument 1010 b by detecting an energized state of the firstinstrument 1010 b with respect to the second instrument 1030 b. Asdiscussed above, each electrosurgical instrument 1010 b, 1030 b has afirst jaw 1012 b, 1032 b and a second jaw 1014 b, 1034 b, and each jaw1012 b, 1014 b, 1032 b, 1034 b has an electrode thereon. The secondinstrument 1030 b monitors the electrodes in its jaws 1032 b, 1034 bwhile the first instrument 1010 b delivers energy to target tissue todetect any change in electrical potential or electrical energy receivedat the second instrument's electrodes to determine if any energy beingdelivered by the first instrument 1010 b to target tissue is reachingthe second instrument 1030 b. If the controller 1050 receives data fromthe second instrument 1030 b indicating a change in the electrodes ofthe second instrument 1030 b while the first instrument 1010 b deliversenergy, the controller 1050 determines a presence of potentiallyunwanted electrical interaction between the first and second instruments1010 b, 1030 b. In response to the detected potentially unwantedelectrical interaction, the system 1000, e.g., the controller 1050thereof, can cause a remedial action to be taken, such as deactivatingthe first instrument 1010 b to stop any potentially unwanted electricalinteraction reducing an amount of power being delivered by the firstinstrument 1010 b to reduce electrical interaction between the first andsecond instruments 1010 b, 1030 b, providing a query to the user, e.g.,via the display 1060 and/or the master control interface 2000, todetermine if the potentially unwanted electrical interaction between thefirst and second instruments 1010 b, 1030 b is acceptable, etc. If theelectrical interaction between the first and second instruments 1010 b,1030 b is acceptable, the surgical procedure can proceed without theenergy delivery of the first instrument 1010 b being stopped or altered.In some embodiments, various thresholds of change in electricalpotential can be set and stored in the memory 1052 so that any changeabove a certain threshold triggers the system 1000, e.g., the controller1050 thereof, as discussed above.

In another embodiment, the second instrument 1030 b has an electricallyisolated end effector, including the first and second jaws 1032 b, 1034b and the electrodes at a distal end of the shaft 1036 b, and theelectrical connection is severable between the end effector and the restof the second instrument 1030 b and the system 1000 through use of anelectrical switch controlled by the controller 1050. The end effectordelivers energy to tissue engaged by the end effector through theelectrode in the first jaw 1032 b, and the controller 1050 monitors theelectrode in the second jaw 1036 b in response to the start of theenergy delivery to detect a change in electrical potential or electricalenergy received at the monitoring electrode that exceeds a predeterminedthreshold (with the predetermined threshold being pre-stored in thememory 1052 for access by the controller 1050). If the threshold isexceeded, the controller 1050 can take one or more different actions,such as selectively shunting excess received energy along a ground pathand/or deactivating (such as by severing the electrical connection) thesecond instrument 1030 b to prevent any further delivery of electricalenergy to the end effector and prevent further transmission of energy,and/or by reducing energy power of the second instrument 1030 b toreduce delivery of electrical energy to the end effector so theelectrical potential of the second instrument 1030 b is below thepredetermined threshold.

In other embodiments, the instruments 1010 b, 1030 b can cooperativelywork together to deliver energy and monitor combined electrical loadsdelivered to tissue to more precisely control energy application andreturn paths. For instance, when one of the instruments 1010 b, 1030 bis positioned on a first side of a tissue wall, and the other instrument1010 b, 1030 b is positioned on a second, opposite side of the tissuewall, one of the instruments 1010 b, 1030 b is configured to apply alevel of energy below a therapeutic level to tissue at the tissue wall.The other instrument 1010 b, 1030 b acts as a return path for theenergy, allowing the instruments 1010, 1030 1010 b, 1030 b to make anumber of determinations about the tissue therebetween, such asimpedance. The instruments 1010 b, 1030 b are also configured tocooperate to apply a therapeutic level of energy to tissue therebetweenat the tissue wall from one or both of the instruments 1010 b, 1030 b,and a return path is used on one or both of the instruments 1010 b, 1030b. When applying energy to tissue at the tissue wall, the instruments1010 b, 1030 b can be located in the same or separate body cavities, andthe instruments 1010 b, 1030 b cooperatively work together to deliverenergy and/or detect energy delivered by the other instrument 1010 b,1030 b to provide more precise energy distribution and application.While examples using mono-polar or bi-polar devices 1010 b, 1030 b havebeen provided, other types of instruments can also be used. Forinstance, a microwave or ultrasound projection device can be arranged ina first body cavity and can be directed toward various probes orfocusing elements that are inserted through a second adjacent bodycavity and placed in between the two cavities. In still another example,one of the instruments can be an endoluminal scope with variouselectrical or ablation contacts on an outer distal surface thereof, andany detected stray energy can allow the scope to be deactivated.

In still other embodiments, the instruments 1010, 1030 are configured towork cooperatively together such that one of the instruments 1010, 1030anchors the other instrument 1010, 1030 and/or tissue while the one ofthe instruments 1010, 1030 providing anchoring is located in either thesame or a separate body cavity to assist in applying force ormaneuvering tissue at a target surgical site, as discussed below.Similar to the embodiments discussed above, in some surgical procedures,various thresholds regarding total and/or individual instrument forceapplied to tissue and/or other instruments can be selected for eachinstrument and/or as a total cumulative amount to avoid any inadvertenttissue trauma.

For example, during some surgical procedures, the first instrument 1010approaches a target surgical site from one side of a tissue wall at thetarget surgical site, and the first instrument 1010 grasps tissue and/orgrasps the second instrument 1030 through the tissue wall to improverigidity and stability of the tissue and/or the second instrument 1030that is interacting with tissue at the target surgical site from anotherside of the tissue wall. In an exemplary surgical procedure, the firstinstrument 1010, for example a larger laparoscopic instrument, is usedto grasp a tissue wall and/or the second instrument 1030, such as asmaller flexible endoscopic instrument, through the tissue wall at atarget surgical site to provide the second instrument 1030 additionalsupport, stability, and/or resistance to loading while the secondinstrument 1030 manipulates tissue at the surgical site directly.Depending on the size and strength of the second instrument 1030, it maynot have the jaw or articulation actuation forces necessary to completea desired surgical task. By allowing the first instrument 1010 tocooperatively connect to the second instrument 1030 either directly tothe second instrument 1030 or indirectly via the tissue wall, the firstinstrument 1010 is able to supply additional force and/or motion to thesecond surgical instrument 1030 and to overcome limitations of thesecond instrument 1030 to result in a larger or different overall forceon tissue than what is possible from each instrument 1010, 1030separately. In other examples, the first instrument 1010 is used incooperation with the second instrument 1030 such that both instruments1010, 1030 grip each other through the tissue wall at the targetsurgical site to improve each instrument's grip on tissue and create alarger, more distributed contact surface between the tissue and theinstruments 1010, 1030 to minimize small tearing of tissue. For example,a flexible endoscopic instrument may be limited to shaft diameters ofapproximately 1.5 to approximately 2.8 mm, while a shaft diameter of alaparoscopic instrument may be from approximately 5 to approximately 12mm, and in some examples the laparoscopic instrument shaft diameter maybe even larger. Thus, as an example, when an approximately 5 mmlaparoscopic instrument, such as a percutaneous laparoscopic sidegrasper or an anvil grasper, grasps an approximately 1.5 mm or anapproximately 3 mm flexible endoscopic instrument, the largerlaparoscopic instrument provides additional support and force to thesmaller flexible endoscopic instrument.

Similar approaches can be taken in a number of different procedures. Inanother embodiment, during a resection of a tumor from a wall of apatient's bladder, the bladder can be limp and non-fluid filled.Locations of the two instruments 1010, 1030 can be coordinated as towhere to grab and distend the bladder. A third instrument can also beintroduced from the laparascopic approach to further adjust forceapplied to bladder tissue to maintain proper distention of the bladder,as needed, allowing the system 1000, e.g., the controller 1050 thereof,to control forces applied by the instruments 1010, 1030 to ensuresuccessful distention while also limiting damage or harm to graspedtissue. In still another embodiment, one of the instruments 1010, 1030includes a balloon that is inflated or deflated to apply more or lessforce on one side of a target surgical site to allow easier dissectionor resection on the other side of the target surgical site and to allowfor a simpler tissue control process because the instruments 1010, 1030work in cooperation to achieve a force on tissue while minimizing anamount of additional work that a user has to perform to monitor theinstruments 1010, 1030 and the force applied to grasped tissue. Thus,the first instrument 1010 in some embodiments does not have sufficientpower or support to complete a function using its own actuationfunctionality, and the second instrument 1030 couples to the firstinstrument 1010 to provide supplemental force, stroke, leverage, etc. toallow the first instrument 1010 to complete a particular function.Various embodiments of surgical instruments including a balloon arefurther described in, for example, previously mentioned U.S. Pat. AppNo. 63/249,980 entitled “Cooperative Access” filed on Sep. 29, 2021. Inanother embodiment, the first instrument 1010 is a stapler that ispositioned and initially clamped onto a target tissue site, but thefirst instrument 1010 requires the second instrument 1030, in the formof a grasper or clamping device, to fully clamp jaws of the firstinstrument 1010 into a closed state onto grasped tissue. After the firstinstrument 1010 is fully closed onto tissue with the help of the secondinstrument 1030, the first instrument 1010 actuates to fire staples intothe tissue. Thus, while the first instrument 1010 is able to actuatestaples on its own, the first instrument 1010 requires the addedclamping force of the second instrument 1030 to fully close.

In some surgical procedures, various combinations of cooperative andopposed/resistive actions, such as forces applied to tissue, can beperformed by each instrument 1010, 1030 to create more preciseinstrument manipulation than is achievable by either instrument 1010,1030 alone. For example, the first and second instruments 1010, 1030 caninitially work first to manipulate tissue in a common direction during afirst part of a surgical procedure, and then the instruments 1010, 1030can work to manipulate tissue in opposite directions in a second part ofthe surgical procedure to improve precision of any applied force whilealso manipulating tissue at a target surgical site to achieve a morebeneficial procedural outcome. In one exemplary surgical procedure, theinstruments 1010, 1030 used in the system 1000 can be surgicalinstruments 1010 c, 1030 c illustrated in FIG. 24 . Each surgicalinstrument 1010 c, 1030 c has a first jaw 1012 c, 1032 c and a secondjaw 1014 c, 1034 c disposed at a distal end of a shaft 1016 c, 1036 c,and each jaw 1012 b, 1014 b, 1032 b, 1034 b has one or more forcesensors thereon. The first and second instruments 1010 c, 1030 c areclamped on common tissue at a target surgical site and act to retract orchange an orientation of the common tissue to gain access to selecttissue and/or to increase visibility of a section of tissue. One or bothof the endoscopes 1020, 1040 monitor movement direction of theinstruments 1010 c, 1030 c, and the one or more force sensors in thesurgical instruments 1010 c, 1030 c monitor force applied to tissue. Assuch, if a movement direction or a force applied to tissue of amonitored instrument 1010 c, 1030 c is changed, the controller 1050detects the change based on visualization from one or both of theendoscopes 1020, 1040 or changes in force detected by the one or moreforce sensors, and the controller 1050 automatically instructs themonitoring instrument 1010 c, 1030 c to perform a change in its behavioror orientation, such as changing a force applied, speed or direction ofmovement, etc., to maintain a cooperative interaction between the firstand second instruments 1010 c, 1030 c and the common tissue. Forexample, the instruments 1010 c, 1030 c may need to control a grippingload applied to target tissue and/or the other instrument 1010 c, 1030 cto avoid or limit inadvertent tissue trauma, in which case a forcethreshold amount can be selected by a user or programmed into the system1000. Using the one or more force sensors in each instrument 1010 c,1030 c, the controller 1050 simultaneously monitors tissue load amounts,such as magnitude and direction, from the instruments 1010 c, 1030 c atthe same time. If the force threshold amount is exceeded on targettissue and/or the other instrument 1010 c, 1030 c, the controller 1050instructs the instruments 1010 c, 1030 c to reduce the gripping load,for example by moving the instruments 1010 c, 1030 c closer together,relaxing a closure force of the jaws of one or both of the instruments1010 c, 1030 c, etc. Use of threshold values can thus ensure that aselectable net tissue load threshold is not exceeded at any point intime at a particular target surgical tissue site. While one or moreforce sensors are discussed herein, a variety of different sensors canbe incorporated into the instruments 1010 c, 1030 c, such as found inU.S. Pat. Pub. No. 2017/0202591 and U.S. Pat. No. 9,757,128,incorporated above.

While use of two instruments 1010 c, 1030 c has been discussed herein,in some embodiments, two instruments 1010 c, 1030 c can act to grasp andmaneuver tissue while a third instrument operates on tissuetherebetween. For example, the first and second instruments 1010 c, 1030c can grasp and spread tissue at a target surgical site while the thirdinstrument operates on or transects target tissue therebetween, and atotal load applied to the tissue by the two instruments 1010 c, 1030 ccan remain constant by moving the instruments 1010 c, 1030 c in responseto the third instrument continuing to operate on the target tissue, ineffect balancing or causing the two instruments 1010 c, 1030 c to resistforces therebetween to maintain a desired force on tissue, for exampleby moving the instruments 1010 c, 1030 c closer together if force beginsto exceed a desired force on the tissue. In other embodiments, two ormore jaws can be incorporated onto the same instrument, and the two ormore jaws function in the same way as the two instruments 1010 c, 1030 cdiscussed above, allowing two jaws on a single instrument to spreadtissue to allow another instrument to access tissue or anatomy betweenthe two jaws.

In addition to altering behavior or functionality of one of theinstruments 1010 c, 1030 c being monitored and similar to thecooperative behavior discussed above, the instrument 1010 c, 1030 cdoing the monitoring can also alter its own behavior and/orfunctionality in response to the monitored instrument 1010 c, 1030 c,similar to the cooperative actions discussed above. For example, themonitoring instrument 1010 c, 1030 c can use monitored behaviors orstatuses of the monitored instrument 1010 c, 1030 c to alter its ownbehavior, such as coordinating its own actions (such as appliedfunctionality, force, rotation about an instrument axis, pivoting aboutan access port, speed, mimicking, shadowing, or mirroring the monitoredinstrument's behavior, etc.) to harmoniously interact with a commontarget surgical site and/or the monitored instrument 1010 c, 1030 c.Additionally, the monitoring instrument 1010 c, 1030 c can act toretract or move tissue to increase visualization by one or both of theendoscopes 1020, 1040 and/or access to a target surgical site for themonitored instrument 1010 c, 1030 c, such as through using variousgrasper devices, suction devices, a cuff-style partial or full wrap ofthe patient's intestine, and/or various endoscopic spreadingvisualization cuffs, etc. The instruments 1010 c, 1030 c can thusmaintain a positional relationship therebetween that can be defined whenthey are coupled, in effect causing the monitoring instrument 1010 c,1030 c to shadow or mimic the monitored instrument 1010 c, 1030 c.

FIG. 25 and FIG. 26 illustrate an exemplary surgical procedure in whicha tumor 3000 is located between a patient's pelvis tissue 3002 and thepatient's colon tissue 3004, and tissue is detached by the secondinstrument 1030 c to improve access to the tumor 3000. The firstinstrument 1010 c used in the system 1000 in this illustrated embodimentis a laparoscopic grasping instrument (considered to be a shadowing orfollowing instrument in the surgical procedure) accessing the targetsurgical site through a laparoscopic approach, and the second instrument1030 c used in the system 1000 in this illustrated embodiment is anendoscopic grasping instrument (considered to be a leading instrument inthe procedure) inserted through a patient's rectum into the patient'scolon. In a first step of the exemplary surgical procedure, the firstinstrument 1010 c retracts tissue at a target surgical site, as shown inFIG. 25 , to allow increased access to the site for dissection by thesecond instrument 1030 c, as shown in FIG. 26 . Numerals 1, 2, 3, and 4are provided in FIG. 25 and FIG. 26 to show sequential movement of theinstruments 1010 c, 1030 c through different surgical positions of theprocedure. A total resistive loading or force relationship on the tissueat the target surgical site is set between the two instruments 1010 c,1030 c, and as the second instrument 1030 c is moved with respect to thetissue at the target surgical site at the tumor 3000, the firstinstrument 1010 c automatically follows movement of the secondinstrument 1030 c while maintaining a similar reactive force based onthe total resistive loading or force relationship rather than stayingstationary and allowing force between the two instruments 1010 c, 1030 cto change. However, the predefined parameters between the twoinstruments 1010 c, 1030 c can be a variety of different values in otherembodiments, such as various maximum or minimum force thresholds, ratesof change of movement such as acceleration or velocity, maximumdisplacement of the shadowed or mimicked behavior, various limits onchanges in orientation between the two instruments, various maximum orminimum overall force loads, maximum or minimum local gripping and/ortear forces on tissue, etc.

A graph in FIG. 27 illustrates resistive load, displacement, andvelocity as possible predefined parameters with threshold values betweena lead device (illustrated as the second surgical device 1030 c in FIG.25 and FIG. 26 ) and a shadowing or mimicking device (illustrated as thefirst surgical device 1010 c in FIG. 25 and FIG. 26 ). FIG. 27illustrates a resistance force load, a displacement value, and avelocity with respect to time for the second instrument 1030 c withinthe colon (labeled as the “First Lead Device” in FIG. 27 ) for asurgical procedure at the colon, and a resistance load force, adisplacement value relative to the pelvic displacement, and a velocityrelative to the pelvic velocity with respect to time for the firstinstrument 1010 c within the pelvic (labeled as the “Second ShadowingMimicking Device” in FIG. 27 ) or a surgical procedure at the pelvis.Additionally, a warning force threshold and a maximum force thresholdare provided for both the first and second instruments 1010 c, 1030 c,at which threshold values a warning is provided to the user and thesystem 1000 prevents additional increases to the resistance load force,respectively. Furthermore, the numerals 1, 2, 3, and 4 that are providedin FIG. 25 and FIG. 26 to show sequential different positions of theinstruments 1010 c, 1030 c during movement are indicated in FIG. 27 toreflect corresponding force, displacement, and velocity at each positionduring movement. For example, as the first and second instruments 1010c, 1030 c move from position 1 to position 2, the resistance load forceon grasped tissue increases until exceeding the warning force threshold,at which point force is reduced by the controller 1050 for bothinstruments 1010 c, 1030 c by slowing down the velocity of the secondinstrument 1030 c (the lead device) while increasing the velocity andreducing the displacement of the first instrument 1010 c (the shadowingdevice) relative to the second instrument 1030 c to move the instruments1010 c, 1030 c closer together. From position 2 to position 3, theresistance load force on grasped tissue exceeds the warning forcethreshold again for the second instrument 1030 c (the lead device), atwhich point the velocity of the second instrument 1030 is lowered by thecontroller 1050 to reduce force. From position 3 to position 4, thelesion completely detaches, at which point the velocity of the secondinstrument 1030 c (the lead device) increases because the patient'spelvis tissue 3002 and the patient's colon tissue 3004 are completelyseparated, and the resistance load force for the first instrument 1010 c(the shadowing device) reaches the maximum force threshold, at whichpoint the displacement of the first instrument 1010 c relative to thesecond instrument 1030 c is reduced by the controller 1050 to move theinstruments 1010 c, 1030 c closer together and reduce the resistanceload force.

While a dissection example is provided above, mimicking, shadowing, ormirroring behavior based on one instrument 1010, 1030 by anotherinstrument 1010, 1030 can involve a number of different procedures. Forexample, in another embodiment, mimicking or shadowing behavior can beused during cooperative tissue suture with two surgical instruments1010, 1030 in which the lead instrument 1030 drives needle entry intotissue while the shadowing or mimicking instrument 1010 grasps aprotruding tip of the needle after it is driven through tissue. Becauseof the shadowing nature of movement between the instruments 1010, 1030,the leading instrument 1030 sets a trajectory for a position at wherethe shadowing instrument 1010 expects the needle to exit tissue.However, a number of different exemplary procedures can use a similarmimicking, shadowing, or mirroring procedure, as provided herein.

One skilled in the art will appreciate further features and advantagesof the invention based on the above-described embodiments. Accordingly,the invention is not to be limited by what has been particularly shownand described, except as indicated by the appended claims. Allpublications and references cited herein are expressly incorporatedherein by reference in their entirety for all purposes.

What is claimed is:
 1. A system, comprising: a first surgical instrumentconfigured to operate on target tissue at a first surgical treatmentsite located within a patient; a second surgical instrument configuredto anchor the first surgical instrument at the first surgical treatmentsite relative to the target tissue; and a controller configured todetermine a total amount of tissue load being applied to the targettissue by the first and second surgical instruments, and to apply limitsto tissue load applied by each of the first and second surgicalinstruments to maintain the total amount of tissue load on the targettissue below a predefined threshold.
 2. The system of claim 1, whereinthe first and second surgical instruments are configured to be disposedwithin a shared body cavity.
 3. The system of claim 1, wherein the firstand second surgical instruments are configured to be disposed onopposite sides of the target tissue in separate body cavities within thepatient.
 4. The system of claim 1, wherein the first and second surgicalinstruments are configured to capture tissue therebetween.
 5. The systemof claim 1, wherein the first and second surgical instruments comprisesurgical staplers, electrosurgical instruments, or graspers.
 6. Thesystem of claim 1, further comprising a flexible endoscope having animage sensor configured to acquire an image of at least one of the firstor second surgical instrument.
 7. The system of claim 6, wherein thecontroller is configured to receive the image from the flexibleendoscope.
 8. A system, comprising: a data processor; and a memorystoring instructions configured to cause the data processor to performoperations comprising: receiving, in real time, data characterizingtissue load being applied by a first surgical instrument to targettissue at a first surgical site within a patient; receiving, in realtime, data characterizing tissue load being applied by a second surgicalinstrument located within the patient, the second surgical instrumentconfigured to anchor the first surgical instrument at the first surgicaltreatment site relative to the target tissue; and determining, based onat least the data received regarding the first and second surgicalinstruments, a total amount of tissue load being applied to the targettissue by the first and second surgical instruments, and applying limitsto tissue load applied by each of the first and second surgicalinstruments on the target tissue to maintain the total amount of tissueload on the target tissue below a predefined threshold.
 9. The system ofclaim 8, wherein the operations of the data processor further comprisesreceiving, in real time, from an image sensor of an endoscope, imagedata characterizing an image of at least one of the first or secondsurgical instruments.
 10. The system of claim 8, wherein the operationsof the a data processor further comprises receiving, in real time, froman image sensor of an endoscope, image data characterizing an image ofat least one of the first or second surgical instruments.
 11. The systemof claim 8, wherein the first and second surgical instruments areconfigured to be disposed within a shared body cavity.
 12. The system ofclaim 8, wherein the first and second surgical instruments areconfigured to be disposed on opposite sides of the target tissue inseparate body cavities within the patient.
 13. The system of claim 8,wherein the first and second surgical instruments are configured tocapture tissue therebetween.
 14. The system of claim 8, wherein thefirst and second surgical instruments comprise surgical staplers,electrosurgical instruments, or graspers.
 15. A method, comprising:receiving, at a controller, in real time with performance of a surgicalprocedure on a patient, data regarding a first surgical instrumentoperating on target tissue at a first surgical treatment site within thepatient; receiving, at the controller, in real time with the performanceof the surgical procedure, data regarding a second surgical instrumentanchoring the first surgical instrument at the first surgical treatmentsite relative to the target tissue; determining, at the controller,based on at least the data received regarding the first and secondsurgical instruments, a total amount of tissue load being applied to thetarget tissue by the first and second surgical instruments; andapplying, using the controller, limits to tissue load applied by each ofthe first and second surgical instruments on the target tissue tomaintain the total amount of tissue load on the target tissue below apredefined threshold.
 16. The method of claim 15, further comprisingaltering, using the controller, the predefined threshold based on alocation of the target tissue within the patient.
 17. The method ofclaim 15, wherein the second surgical instrument provides the dataregarding the first and second surgical instruments to the controller.18. The method of claim 15, wherein the first and second surgicalinstruments comprise surgical staplers, electrosurgical instruments, orgraspers.
 19. The method of claim 15, further comprising receiving, atthe controller from a flexible endoscope having an image sensor, imagedata characterizing an image of at least one of the first or secondsurgical instruments.